From patchwork Sat Dec 31 21:45:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF489C4167B for ; Sat, 31 Dec 2022 21:46:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9709C8E0007; Sat, 31 Dec 2022 16:46:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 80C5D8E0001; Sat, 31 Dec 2022 16:46:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60F308E0007; Sat, 31 Dec 2022 16:46:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 39FF88E0001 for ; Sat, 31 Dec 2022 16:46:12 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A689E160270 for ; Sat, 31 Dec 2022 21:46:11 +0000 (UTC) X-FDA: 80303934942.27.4CACC0E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 11B44120004 for ; Sat, 31 Dec 2022 21:46:09 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=B9hqsg2S; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523170; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ey9MkIG/Wma0AYrd9YHtW6VqfAs3Lwkb3yAjavmqz0o=; b=1khLHtVuaPeI4ieBEz2QV3u3teYd47jZ2ER4VPdU9Ezbsu0aLzvNBoJZD1GS+TuPrkSr7s 4gHJkQwOgbpHcodqWSntWVx6732041RdYH448FzAyH+VQ6tUo4JazDIyc2Uiqw8l0Zkxfs hlz7akO+xBNN6aGklo7ndOClN4ZyhQY= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=B9hqsg2S; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523170; a=rsa-sha256; cv=none; b=W81F34NmP/VJl4zZojys9hJyID5yl9n3V8vNCdBkfHrM3XH5uPQRdkl/wRlmAo2gm6+D4n cOzkYGn2h8oIRbHQN4Nxflt4wPr5KduAv8wUbUd3gcSnZQYD3z51Z9QxGyF0Cld5UPVu0V kAZJW90R/hOMbTtWSxzU47puzLwJtkA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Ey9MkIG/Wma0AYrd9YHtW6VqfAs3Lwkb3yAjavmqz0o=; b=B9hqsg2Sn9buqgZYYVvq/J1E8X xGl5P9Ww9QWoYmoMytOL1a8laAd9CUF04LnzVTYr2XjaahjZzRD4Ggv8MZef2RzgynsPWprlBLmxX oB9dO06Q2xNfhwxoO6PCqBL8as98zSjprSBtIgjv+SePJgA8RC37IpzlxYDTCJtFz2Y5oaOTQpbmr 1yrptZSzhHy39aGnyZW0m8lxWogsg09skHNyTKta/DjQ5MX6cwl6AjvH9rLuXg846j0QC4uHTBvB/ 0MTII12hjmEfjhUHUYNmAuunHzNo2x+64R4K3WhrLcQwLA7ya9hUeapTIUHX+uzFFkFUl9AwaIHWZ prEd11iA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgC-00BkaT-Oq; Sat, 31 Dec 2022 21:46:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 01/22] mm: Remove folio_pincount_ptr() and head_compound_pincount() Date: Sat, 31 Dec 2022 21:45:49 +0000 Message-Id: <20221231214610.2800682-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 11B44120004 X-Stat-Signature: e11xja1smwgw51nbr3xegweg3j55dxun X-Rspam-User: X-HE-Tag: 1672523169-853726 X-HE-Meta: U2FsdGVkX19m+m5i/QGn1Rphp8nYhflK1GT34C2B4aheBHdiMX5sp/VrJ7+bWOEwjTAFll6PuxrYa5raGRnHrlnmhiDnCmZufGPprTdqWdhlsZlXKyeDgX+QLKdV3AQqZ2sW2j50N+xKxXbSv8punXUoyNnwyMeIFxlqXWidMuWw0ONdLB9P/NRu6hbfql+jLkvLjbPs3SNArS6Ad+ODsUfNwiu3yoEnRnmBy9EtELZTbfT+CPI/GOPrwsKzP+2D/RVwWoXf/LAw5hOeALnEpDk5QY+viHN75lHHejha9tMWMwp1WSd330n7s8ky9fqJ3I0pyclvBpNaS+p/WR6UqbZVzklMhCCvk626EciITHWCgI45Txg8yCVFi1s9Hvz2eeWkPHf93fR/hGsW7RImtdSDJSDPyb5GyOqdr9ZEvfq2a/dcU2IMXMEAD2L7tbWPMxcH2+rvDeFYstc6606nIoXOKhrBTNh/wViDdnExDC7FA6XSbpyAMFDDgxKdGB+mqVx/NOFlv2EnIVJuhfbPUJ4niI93rYWcf3EbRQfnqluI/gN3JBZwvUDvXf1yyKzjFb16OH4iQrykgTany/aiTA1NQ12hjNfaFLX6THyw327tPWktPISq8KT2Lvw8QwE4fjwzF9WP6itq58KRV9GaPnOQVLGVHyuNV3W5QgHGrhZIA8JwJA6jTjBXGy9n3kYg+Xrl3OrhyvJdkY3wWb6UEo9oqCOw9L8xllZI3pQGRFuQ1S2NYSDCexUX8Ak2KPSTvUbZ+UwUII6mONBcy/Me9hBoDlkR6G48yKiRwhgZdkQhAy70U8EYuDhBO5qyWozsyQeoLvFMmQgqQx4CcihsY//WWlS1nIAAqsOPrR8hMRjGtL4rnLgpvW0zw1BcgF2Ujfk3Tl4hK/nyyeBeTfxUudzYm3xpr5ZIWSpEqUiRp2weFxh69uZTNw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can use folio->_pincount directly, since all users are guarded by tests of compound/large. Signed-off-by: Matthew Wilcox (Oracle) --- Documentation/core-api/pin_user_pages.rst | 29 +++++++++++------------ include/linux/mm.h | 14 ++--------- include/linux/mm_types.h | 5 ---- mm/debug.c | 4 ++-- mm/gup.c | 8 +++---- mm/huge_memory.c | 4 ++-- mm/hugetlb.c | 4 ++-- mm/page_alloc.c | 9 ++++--- 8 files changed, 32 insertions(+), 45 deletions(-) diff --git a/Documentation/core-api/pin_user_pages.rst b/Documentation/core-api/pin_user_pages.rst index b18416f4500f..674edf62f186 100644 --- a/Documentation/core-api/pin_user_pages.rst +++ b/Documentation/core-api/pin_user_pages.rst @@ -55,18 +55,17 @@ flags the caller provides. The caller is required to pass in a non-null struct pages* array, and the function then pins pages by incrementing each by a special value: GUP_PIN_COUNTING_BIAS. -For compound pages, the GUP_PIN_COUNTING_BIAS scheme is not used. Instead, -an exact form of pin counting is achieved, by using the 2nd struct page -in the compound page. A new struct page field, compound_pincount, has -been added in order to support this. - -This approach for compound pages avoids the counting upper limit problems that -are discussed below. Those limitations would have been aggravated severely by -huge pages, because each tail page adds a refcount to the head page. And in -fact, testing revealed that, without a separate compound_pincount field, -page overflows were seen in some huge page stress tests. - -This also means that huge pages and compound pages do not suffer +For large folios, the GUP_PIN_COUNTING_BIAS scheme is not used. Instead, +the extra space available in the struct folio is used to store the +pincount directly. + +This approach for large folios avoids the counting upper limit problems +that are discussed below. Those limitations would have been aggravated +severely by huge pages, because each tail page adds a refcount to the +head page. And in fact, testing revealed that, without a separate pincount +field, refcount overflows were seen in some huge page stress tests. + +This also means that huge pages and large folios do not suffer from the false positives problem that is mentioned below.:: Function @@ -264,9 +263,9 @@ place.) Other diagnostics ================= -dump_page() has been enhanced slightly, to handle these new counting -fields, and to better report on compound pages in general. Specifically, -for compound pages, the exact (compound_pincount) pincount is reported. +dump_page() has been enhanced slightly to handle these new counting +fields, and to better report on large folios in general. Specifically, +for large folios, the exact pincount is reported. References ========== diff --git a/include/linux/mm.h b/include/linux/mm.h index f3f196e4d66d..ec801f24ef61 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1006,11 +1006,6 @@ static inline void folio_set_compound_dtor(struct folio *folio, void destroy_large_folio(struct folio *folio); -static inline int head_compound_pincount(struct page *head) -{ - return atomic_read(compound_pincount_ptr(head)); -} - static inline void set_compound_order(struct page *page, unsigned int order) { page[1].compound_order = order; @@ -1637,11 +1632,6 @@ static inline struct folio *pfn_folio(unsigned long pfn) return page_folio(pfn_to_page(pfn)); } -static inline atomic_t *folio_pincount_ptr(struct folio *folio) -{ - return &folio_page(folio, 1)->compound_pincount; -} - /** * folio_maybe_dma_pinned - Report if a folio may be pinned for DMA. * @folio: The folio. @@ -1659,7 +1649,7 @@ static inline atomic_t *folio_pincount_ptr(struct folio *folio) * expected to be able to deal gracefully with a false positive. * * For large folios, the result will be exactly correct. That's because - * we have more tracking data available: the compound_pincount is used + * we have more tracking data available: the _pincount field is used * instead of the GUP_PIN_COUNTING_BIAS scheme. * * For more information, please see Documentation/core-api/pin_user_pages.rst. @@ -1670,7 +1660,7 @@ static inline atomic_t *folio_pincount_ptr(struct folio *folio) static inline bool folio_maybe_dma_pinned(struct folio *folio) { if (folio_test_large(folio)) - return atomic_read(folio_pincount_ptr(folio)) > 0; + return atomic_read(&folio->_pincount) > 0; /* * folio_ref_count() is signed. If that refcount overflows, then diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3b8475007734..5d9bf1f79e96 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -443,11 +443,6 @@ static inline atomic_t *subpages_mapcount_ptr(struct page *page) return &page[1].subpages_mapcount; } -static inline atomic_t *compound_pincount_ptr(struct page *page) -{ - return &page[1].compound_pincount; -} - /* * Used for sizing the vmemmap region on some architectures */ diff --git a/mm/debug.c b/mm/debug.c index 7f8e5f744e42..893c9dbf76ca 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -94,11 +94,11 @@ static void __dump_page(struct page *page) page, page_ref_count(head), mapcount, mapping, page_to_pgoff(page), page_to_pfn(page)); if (compound) { - pr_warn("head:%p order:%u compound_mapcount:%d subpages_mapcount:%d compound_pincount:%d\n", + pr_warn("head:%p order:%u compound_mapcount:%d subpages_mapcount:%d pincount:%d\n", head, compound_order(head), head_compound_mapcount(head), head_subpages_mapcount(head), - head_compound_pincount(head)); + atomic_read(&folio->_pincount)); } #ifdef CONFIG_MEMCG diff --git a/mm/gup.c b/mm/gup.c index f45a3a5be53a..38ba1697dd61 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -111,7 +111,7 @@ static inline struct folio *try_get_folio(struct page *page, int refs) * FOLL_GET: folio's refcount will be incremented by @refs. * * FOLL_PIN on large folios: folio's refcount will be incremented by - * @refs, and its compound_pincount will be incremented by @refs. + * @refs, and its pincount will be incremented by @refs. * * FOLL_PIN on single-page folios: folio's refcount will be incremented by * @refs * GUP_PIN_COUNTING_BIAS. @@ -157,7 +157,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) * try_get_folio() is left intact. */ if (folio_test_large(folio)) - atomic_add(refs, folio_pincount_ptr(folio)); + atomic_add(refs, &folio->_pincount); else folio_ref_add(folio, refs * (GUP_PIN_COUNTING_BIAS - 1)); @@ -182,7 +182,7 @@ static void gup_put_folio(struct folio *folio, int refs, unsigned int flags) if (flags & FOLL_PIN) { node_stat_mod_folio(folio, NR_FOLL_PIN_RELEASED, refs); if (folio_test_large(folio)) - atomic_sub(refs, folio_pincount_ptr(folio)); + atomic_sub(refs, &folio->_pincount); else refs *= GUP_PIN_COUNTING_BIAS; } @@ -232,7 +232,7 @@ int __must_check try_grab_page(struct page *page, unsigned int flags) */ if (folio_test_large(folio)) { folio_ref_add(folio, 1); - atomic_add(1, folio_pincount_ptr(folio)); + atomic_add(1, &folio->_pincount); } else { folio_ref_add(folio, GUP_PIN_COUNTING_BIAS); } diff --git a/mm/huge_memory.c b/mm/huge_memory.c index abe6cfd92ffa..ca2eaec84726 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2479,9 +2479,9 @@ static void __split_huge_page_tail(struct page *head, int tail, * of swap cache pages that store the swp_entry_t in tail pages. * Fix up and warn once if private is unexpectedly set. * - * What of 32-bit systems, on which head[1].compound_pincount overlays + * What of 32-bit systems, on which folio->_pincount overlays * head[1].private? No problem: THP_SWAP is not enabled on 32-bit, and - * compound_pincount must be 0 for folio_ref_freeze() to have succeeded. + * pincount must be 0 for folio_ref_freeze() to have succeeded. */ if (!folio_test_swapcache(page_folio(head))) { VM_WARN_ON_ONCE_PAGE(page_tail->private != 0, page_tail); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index db895230ee7e..c01493ceeb8d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1480,7 +1480,7 @@ static void __destroy_compound_gigantic_folio(struct folio *folio, atomic_set(folio_mapcount_ptr(folio), 0); atomic_set(folio_subpages_mapcount_ptr(folio), 0); - atomic_set(folio_pincount_ptr(folio), 0); + atomic_set(&folio->_pincount, 0); for (i = 1; i < nr_pages; i++) { p = folio_page(folio, i); @@ -2002,7 +2002,7 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, } atomic_set(folio_mapcount_ptr(folio), -1); atomic_set(folio_subpages_mapcount_ptr(folio), 0); - atomic_set(folio_pincount_ptr(folio), 0); + atomic_set(&folio->_pincount, 0); return true; out_error: diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0745aedebb37..a04ed7f72b36 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -775,11 +775,13 @@ void free_compound_page(struct page *page) static void prep_compound_head(struct page *page, unsigned int order) { + struct folio *folio = (struct folio *)page; + set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); set_compound_order(page, order); atomic_set(compound_mapcount_ptr(page), -1); atomic_set(subpages_mapcount_ptr(page), 0); - atomic_set(compound_pincount_ptr(page), 0); + atomic_set(&folio->_pincount, 0); } static void prep_compound_tail(struct page *head, int tail_idx) @@ -1291,6 +1293,7 @@ static inline bool free_page_is_bad(struct page *page) static int free_tail_pages_check(struct page *head_page, struct page *page) { + struct folio *folio = (struct folio *)head_page; int ret = 1; /* @@ -1314,8 +1317,8 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) bad_page(page, "nonzero subpages_mapcount"); goto out; } - if (unlikely(head_compound_pincount(head_page))) { - bad_page(page, "nonzero compound_pincount"); + if (unlikely(atomic_read(&folio->_pincount))) { + bad_page(page, "nonzero pincount"); goto out; } break; From patchwork Sat Dec 31 21:45:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086172 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65532C3DA7D for ; Sat, 31 Dec 2022 21:46:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CBC88E0009; Sat, 31 Dec 2022 16:46:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DEF38E0001; Sat, 31 Dec 2022 16:46:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E72FA8E0009; Sat, 31 Dec 2022 16:46:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C24888E0001 for ; Sat, 31 Dec 2022 16:46:13 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 7C4714048C for ; Sat, 31 Dec 2022 21:46:13 +0000 (UTC) X-FDA: 80303935026.12.09E70A9 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id D85C412000C for ; Sat, 31 Dec 2022 21:46:11 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="jo78/JPD"; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523172; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L52jAtJFxk47TjLEfZqT2z6NTyto8MQVK1ht6ImQE4I=; b=Bfvhuzw0kVbUsbzdwKqKVKlH2dJ5hU1EuaDSZ94aQZ5EppmS3UzIV6dw4WRG3kD7gxfjjg mbgFmklSeOIvuGBOv45WfWBAhHXjZYCDF90N7l9f5boda3Nz7Q0OJc/pTElSDeCi2bqUir Mynxdzm/GeCJ1qyTaY6yHS7chYmofjE= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="jo78/JPD"; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523172; a=rsa-sha256; cv=none; b=K0vEgLKwSxTdmE+LDuuWhj2gboDnqiYPGbe2qd8Up7O4Hv7yHRU4LTnOLSvcRLZJ2O+EKm 5auGvxM+vKTuDYs6A+XgBr39nHppWwBBTaMTOULRPQ+e6xWoZC/tJO1QDMXxytNykKi16d BmzKnNt0frzd97ljl528ISEbnwrKreo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=L52jAtJFxk47TjLEfZqT2z6NTyto8MQVK1ht6ImQE4I=; b=jo78/JPDByiXH3c5BPL5hp8McM VmiihabBWHZKNLxSm5skqm5qfFqVM0iD8lkPOq2NiiIdynAFGWM8f65LGJfcxH0W8LyAEk3LDPIHh Cx5OJLh5egXIBhjmS3zJ0775xgYLpzIkoMOLalg6p+EIjTVAvaTVcQ3aBC6jmlhQEAkoQ70/mxIM6 4mW120CeUNOUEJhM9w2yC8rpIkEOUpcBl1B9b1x0LZnPmuwoLcyggYOV3H8Iw9c/7cRntsKKgs+yq WMK+bavbDkpIhdxKBMwywcTpF3ES+x3knHBAXToPpU8TR4S9vjtQs8SWf7IqcZVZ5DZuMPGESOmdo qukPBm+A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgC-00BkaV-Ro; Sat, 31 Dec 2022 21:46:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 02/22] mm: Convert head_subpages_mapcount() into folio_nr_pages_mapped() Date: Sat, 31 Dec 2022 21:45:50 +0000 Message-Id: <20221231214610.2800682-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: D85C412000C X-Stat-Signature: pporfoopjwwfn3famtr78x11puz8aubi X-Rspam-User: X-HE-Tag: 1672523171-60135 X-HE-Meta: U2FsdGVkX1+2W3NS0T/6B7Amn9MSeboClsTdxSmavb7ulu+JK6aN3kxnwpQ+poat2xyok9G/1Qix9roIzVMYCuudkanYASziM7ElinPaWIA9Ov8yPByU/lF1thnRkZRZE+t2X3RAoHrR7G+EtyZk9Mg5sx5PYJBbOR7WrD8zATCmW0hBNjIJ/2kSBhDjb3EjyNxopK6Ng/mBZdWy+3KIvam2kWHE6AXHvRyOh0C3RHsfZIjlenHXNYgrLuLIUdFjGMxko9LAf+V0G+eaXO5RXbxcBYi5oCliKlAyMQmIMbnwHYiLeZsYFqcEJLJO8OBS1eUszyUhY7f8AkKCP5Q/HjYXI9VBRfgf4TgyZ1HtrX1eJWskn7LJvW+W/N4+TIci9vq15OYER2M4B+x5rDmNzhTxVqXoaXCvO4nvtBFRfzdxDJONyWai1J5LubEJPOU+ndPmUygZiFOxy7gmDKJZJuLnR7I7jBWKbezaMgNGSUMawy4O4aGXY2lsmEfxYoX6RSjnL6oGwDmP+McwtkRMM1qgqTD1c2YCxxCjLQTibg03gz55Tgb8lVy3HwVeHjJXQhWgk8Bpgnnd06+cbGogczNCBDAuENIZYNYL+ASCaA9nwEa+ut+et3CdpLV54Et92b1NBFqgRoANtNjk8ogL5ZY7ZIAhU8IUJLf32W6fnTJzBeTnWMH9OA8RYltkjt9lJQidmGasv0Jc3ZaXQAqFEPq8WQh9nHLaZlD7DgApFPHrhI8oCIyB/kscGr/pMmLeJ1ssnSud4iuzwvfD653amPrP0C9h+5f5qFA/knJg1oDp21M+nGwp3jT3tut+jBv0vnDpr0YnTPCOs6t324jg9xHZlC3uTrrWNlY4Aww9nJiveBfsdZwiekC7Bui/g62Y3E2zOXbQMaE54F1+4IyZqpvYg1CrRIb8IRA4c0nPXFmXVsiI7/5wAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Calling this 'mapcount' is confusing since mapcount is usually the number of times something is mapped; instead this is the number of mapped pages. It's also better to enforce that this is a folio rather than a head page. Move folio_nr_pages_mapped() into mm/internal.h since this is not something we want device drivers or filesystems poking at. Get rid of folio_subpages_mapcount_ptr() and use folio->_nr_pages_mapped directly. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 22 ++-------------------- include/linux/mm_types.h | 12 +++--------- mm/debug.c | 4 ++-- mm/hugetlb.c | 4 ++-- mm/internal.h | 18 ++++++++++++++++++ mm/rmap.c | 9 +++++---- 6 files changed, 32 insertions(+), 37 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ec801f24ef61..7ee1938278f5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -838,24 +838,6 @@ static inline int head_compound_mapcount(struct page *head) return atomic_read(compound_mapcount_ptr(head)) + 1; } -/* - * If a 16GB hugetlb page were mapped by PTEs of all of its 4kB sub-pages, - * its subpages_mapcount would be 0x400000: choose the COMPOUND_MAPPED bit - * above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). Hugetlb currently - * leaves subpages_mapcount at 0, but avoid surprise if it participates later. - */ -#define COMPOUND_MAPPED 0x800000 -#define SUBPAGES_MAPPED (COMPOUND_MAPPED - 1) - -/* - * Number of sub-pages mapped by PTE, does not include compound mapcount. - * Must be called only on head of compound page. - */ -static inline int head_subpages_mapcount(struct page *head) -{ - return atomic_read(subpages_mapcount_ptr(head)) & SUBPAGES_MAPPED; -} - /* * The atomic page->_mapcount, starts from -1: so that transitions * both from it and to it can be tracked, using atomic_inc_and_test @@ -915,9 +897,9 @@ static inline bool folio_large_is_mapped(struct folio *folio) { /* * Reading folio_mapcount_ptr() below could be omitted if hugetlb - * participated in incrementing subpages_mapcount when compound mapped. + * participated in incrementing nr_pages_mapped when compound mapped. */ - return atomic_read(folio_subpages_mapcount_ptr(folio)) > 0 || + return atomic_read(&folio->_nr_pages_mapped) > 0 || atomic_read(folio_mapcount_ptr(folio)) >= 0; } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5d9bf1f79e96..fc44d5bab7b8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -307,7 +307,7 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) * @_folio_dtor: Which destructor to use for this folio. * @_folio_order: Do not use directly, call folio_order(). * @_compound_mapcount: Do not use directly, call folio_entire_mapcount(). - * @_subpages_mapcount: Do not use directly, call folio_mapcount(). + * @_nr_pages_mapped: Do not use directly, call folio_mapcount(). * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). * @_flags_2: For alignment. Do not use. @@ -361,7 +361,7 @@ struct folio { unsigned char _folio_dtor; unsigned char _folio_order; atomic_t _compound_mapcount; - atomic_t _subpages_mapcount; + atomic_t _nr_pages_mapped; atomic_t _pincount; #ifdef CONFIG_64BIT unsigned int _folio_nr_pages; @@ -404,7 +404,7 @@ FOLIO_MATCH(compound_head, _head_1); FOLIO_MATCH(compound_dtor, _folio_dtor); FOLIO_MATCH(compound_order, _folio_order); FOLIO_MATCH(compound_mapcount, _compound_mapcount); -FOLIO_MATCH(subpages_mapcount, _subpages_mapcount); +FOLIO_MATCH(subpages_mapcount, _nr_pages_mapped); FOLIO_MATCH(compound_pincount, _pincount); #ifdef CONFIG_64BIT FOLIO_MATCH(compound_nr, _folio_nr_pages); @@ -427,12 +427,6 @@ static inline atomic_t *folio_mapcount_ptr(struct folio *folio) return &tail->compound_mapcount; } -static inline atomic_t *folio_subpages_mapcount_ptr(struct folio *folio) -{ - struct page *tail = &folio->page + 1; - return &tail->subpages_mapcount; -} - static inline atomic_t *compound_mapcount_ptr(struct page *page) { return &page[1].compound_mapcount; diff --git a/mm/debug.c b/mm/debug.c index 893c9dbf76ca..8e58e8dab0b2 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -94,10 +94,10 @@ static void __dump_page(struct page *page) page, page_ref_count(head), mapcount, mapping, page_to_pgoff(page), page_to_pfn(page)); if (compound) { - pr_warn("head:%p order:%u compound_mapcount:%d subpages_mapcount:%d pincount:%d\n", + pr_warn("head:%p order:%u compound_mapcount:%d nr_pages_mapped:%d pincount:%d\n", head, compound_order(head), head_compound_mapcount(head), - head_subpages_mapcount(head), + folio_nr_pages_mapped(folio), atomic_read(&folio->_pincount)); } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c01493ceeb8d..55e744abb962 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1479,7 +1479,7 @@ static void __destroy_compound_gigantic_folio(struct folio *folio, struct page *p; atomic_set(folio_mapcount_ptr(folio), 0); - atomic_set(folio_subpages_mapcount_ptr(folio), 0); + atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); for (i = 1; i < nr_pages; i++) { @@ -2001,7 +2001,7 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, set_compound_head(p, &folio->page); } atomic_set(folio_mapcount_ptr(folio), -1); - atomic_set(folio_subpages_mapcount_ptr(folio), 0); + atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); return true; diff --git a/mm/internal.h b/mm/internal.h index bcf75a8b032d..f3bb12e77980 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -52,6 +52,24 @@ struct folio_batch; void page_writeback_init(void); +/* + * If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages, + * its nr_pages_mapped would be 0x400000: choose the COMPOUND_MAPPED bit + * above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). Hugetlb currently + * leaves nr_pages_mapped at 0, but avoid surprise if it participates later. + */ +#define COMPOUND_MAPPED 0x800000 +#define FOLIO_PAGES_MAPPED (COMPOUND_MAPPED - 1) + +/* + * How many individual pages have an elevated _mapcount. Excludes + * the folio's entire_mapcount. + */ +static inline int folio_nr_pages_mapped(struct folio *folio) +{ + return atomic_read(&folio->_nr_pages_mapped) & FOLIO_PAGES_MAPPED; +} + static inline void *folio_raw_mapping(struct folio *folio) { unsigned long mapping = (unsigned long)folio->mapping; diff --git a/mm/rmap.c b/mm/rmap.c index b616870a09be..09f4d260a46c 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1087,12 +1087,13 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, int total_compound_mapcount(struct page *head) { + struct folio *folio = (struct folio *)head; int mapcount = head_compound_mapcount(head); int nr_subpages; int i; /* In the common case, avoid the loop when no subpages mapped by PTE */ - if (head_subpages_mapcount(head) == 0) + if (folio_nr_pages_mapped(folio) == 0) return mapcount; /* * Add all the PTE mappings of those subpages mapped by PTE. @@ -1243,7 +1244,7 @@ void page_add_anon_rmap(struct page *page, nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { nr_pmdmapped = thp_nr_pages(page); - nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) nr = 0; @@ -1349,7 +1350,7 @@ void page_add_file_rmap(struct page *page, nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { nr_pmdmapped = thp_nr_pages(page); - nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) nr = 0; @@ -1414,7 +1415,7 @@ void page_remove_rmap(struct page *page, nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED)) { nr_pmdmapped = thp_nr_pages(page); - nr = nr_pmdmapped - (nr & SUBPAGES_MAPPED); + nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of another remove and an add? */ if (unlikely(nr < 0)) nr = 0; From patchwork Sat Dec 31 21:45:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086174 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF9BFC4167B for ; Sat, 31 Dec 2022 21:46:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB4F78E000B; Sat, 31 Dec 2022 16:46:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B3F9A8E0001; Sat, 31 Dec 2022 16:46:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B9088E000B; Sat, 31 Dec 2022 16:46:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6480B8E0001 for ; Sat, 31 Dec 2022 16:46:16 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3892F1A05A5 for ; Sat, 31 Dec 2022 21:46:16 +0000 (UTC) X-FDA: 80303935152.01.213DB4C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id AA7A8120003 for ; Sat, 31 Dec 2022 21:46:13 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=i57N+Hwh; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523173; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WvVzapyi0dCw+kNmNHNzQOI1Ac8jKfCSQjkrQYngcng=; b=77H6Q4dt+ZUoMQWD1/PUeQ/kdsC/FLd3rz8yD/J+ef/rRbIPVPsy/+wuRW+Nu3rRHQdh5U xEURaJzUjGmJYPLz6YVTrwjQucoueAUHnf0unaE/76yFgjWf5JhJpHk4EBqEsftm4DoLEh BWhek30w/An2UfcrbPHPEl2YgwbLhtY= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=i57N+Hwh; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523173; a=rsa-sha256; cv=none; b=c/mD96FaVAxSRI2zIhGwScAv3K+BULTzPhCU1vG68HzPuU4JzE9zQ5EqlxGHQlr9W1tK6a xcJzVzZwEZ0orPnjzi7nJ1Ybp9ttnKE+8XQOk6Gp4nu5ti9didFxrB4nv/mFGgzeUtDg0D ilsU6XlOimD4bE03o4W33jKKjhvsj2I= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=WvVzapyi0dCw+kNmNHNzQOI1Ac8jKfCSQjkrQYngcng=; b=i57N+Hwh8G7gZw4PSLejBH0ZDd EQc/mMknBgINjjrvGOydPCiMXLYzP3dHou2IQ/C9w+4J08fsBP2l6wZt5y7OAqIcWnYUqiRDk0koY 9RL3xo3st0GTcu+BTgB3rNwSJkRL1fTeHX6txX1gFPKMszx0nSU7Jq2+8df6ScROs4mz7njMRqXDX wNrSozQIF+LC/VfaEk9kTC021IDuzsCPYg4+a2dT94w3IMrSDcBibuzn30pIsovIyeDbdlzttdRet 4JdyvP0kl2Ucew4zyLz6Fi1k3Giqq9um2ZZLUQgNAT0xW9lX0VmuQAy87BXJbh8AcCdppVpMgOqwF UIPVRTPw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgC-00BkaX-UT; Sat, 31 Dec 2022 21:46:12 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 03/22] doc: Clarify refcount section by referring to folios & pages Date: Sat, 31 Dec 2022 21:45:51 +0000 Message-Id: <20221231214610.2800682-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: AA7A8120003 X-Stat-Signature: hmzzntarnb5psrag3g44btp8461ptunz X-Rspam-User: X-HE-Tag: 1672523173-489429 X-HE-Meta: U2FsdGVkX18WJFvFYQgnCi52ve+8l+LjpqERuZWxkVXelVA3bUylE3nHJSXV1sB4ceBNFzchiZt4qBMLVDPXLjnH3m/pU2TR7WXsmU38hyeb54HYi/BxhGw8oq0wiXc+bEXjqxGpdnQcz8omjMBFTAkDukVGSquIYozPgleCw6FuHBn4rIZ3peYKy5NuuKlagTNcn0i6letv8uyzXUB8HrXTgKwfh2yXSuQ9YQSU/Mo7V+teQHWNUeJzdqg27gCDp+LO96gZACxTbYertoe/D3cPyR12p8FT3qd1W6S4IozfWnR9YUoUbqUunH/9Qby7eru2QpF75b4HBNbSqwHmzQnu2vOlB3I+wOJl0d9Ipa/b9kFXUnKfTMC/TIHTXt3d6TS0sS5VH/J2IyF4tOfmGCUIe5hFZcF38qmJF6w3dS9Np9wxyIThX8xB8vm8ZvAbyFZnHt65oZLmQwQVv8nCyUODuSk1jzNUMOK5NAdL/hpUGdOYvTcbI8f4BcZ/VY/9cUkI/e/tothshudRufdjFg3c2L8kvhC0VylYJa2G6F0p+wIneMvf5onC7nTQWSfUeX4fFFbkE4K5vHgLnzaHBXuHpNUnCW17MnWxxPR5iCIvMt42ftKAsucKvxa8lQ4Uh3jxujBMnAONjZF3Z82jEtTzx3M9OSxsIxECPiF5dc0a+KIv6FeWxKgeCFykkTQ/ZVyP4Y+Hf6Ku/g5Erp4C/sjirYZKxA6G0HIso+SeyI/lDGsSfM7Xqs6eg8MPWQiEPRc+tWTmqHHdXOFRCN+a7OG7T1Q7uKCmavrgWPtM+D7jzi8/KtrXbrY0mP1+9KDpSmwam38enUCWRGNefAlaDKA2zGuraTnLKR9dqzVKBPHSIuoCeA8DMSBbxOdWzfO3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Include the rename of subpages_mapcount to _nr_pages_mapped. Signed-off-by: Matthew Wilcox (Oracle) --- Documentation/mm/transhuge.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index ec3dc5b04226..03bbd0a19041 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -112,20 +112,20 @@ Refcounts and transparent huge pages Refcounting on THP is mostly consistent with refcounting on other compound pages: - - get_page()/put_page() and GUP operate on head page's ->_refcount. + - get_page()/put_page() and GUP operate on the folio->_refcount. - ->_refcount in tail pages is always zero: get_page_unless_zero() never succeeds on tail pages. - - map/unmap of PMD entry for the whole compound page increment/decrement - ->compound_mapcount, stored in the first tail page of the compound page; - and also increment/decrement ->subpages_mapcount (also in the first tail) - by COMPOUND_MAPPED when compound_mapcount goes from -1 to 0 or 0 to -1. + - map/unmap of a PMD entry for the whole THP increment/decrement + folio->_entire_mapcount and also increment/decrement + folio->_nr_pages_mapped by COMPOUND_MAPPED when _entire_mapcount + goes from -1 to 0 or 0 to -1. - - map/unmap of sub-pages with PTE entry increment/decrement ->_mapcount - on relevant sub-page of the compound page, and also increment/decrement - ->subpages_mapcount, stored in first tail page of the compound page, when - _mapcount goes from -1 to 0 or 0 to -1: counting sub-pages mapped by PTE. + - map/unmap of individual pages with PTE entry increment/decrement + page->_mapcount and also increment/decrement folio->_nr_pages_mapped + when page->_mapcount goes from -1 to 0 or 0 to -1 as this counts + the number of pages mapped by PTE. split_huge_page internally has to distribute the refcounts in the head page to the tail pages before clearing all PG_head/tail bits from the page From patchwork Sat Dec 31 21:45:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71B84C4167B for ; Sat, 31 Dec 2022 21:46:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 44E4E8E0014; Sat, 31 Dec 2022 16:46:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 249068E0018; Sat, 31 Dec 2022 16:46:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F35738E0016; Sat, 31 Dec 2022 16:46:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D54538E0001 for ; Sat, 31 Dec 2022 16:46:36 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B82ADA02F8 for ; Sat, 31 Dec 2022 21:46:36 +0000 (UTC) X-FDA: 80303935992.22.717EC64 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf09.hostedemail.com (Postfix) with ESMTP id 3FC1814000D for ; Sat, 31 Dec 2022 21:46:34 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TUtKBu96; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523195; a=rsa-sha256; cv=none; b=dT67iXGoVC9+nnd2UmBSS7bYQPY+OrW25mHFYDCgM7L7SfqqIhc+JXWXvrwMQrIAmdjOVe 8HGbciYjy/hu9juDou81BWVYLetoim2cgvSj/IWn4capD63UXBULQfQxHtY1UPBAtTwTxV vKxk8G38COAOo/8tXY4dgQX6Dil2aCg= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=TUtKBu96; spf=none (imf09.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523195; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QgsvB1oN/cnvlW92dNAD5N+075D0L1/Jt0SsBfnWtRI=; b=7Us0r5NDlYNiOpmFYIaooMhLgTxip0WnNtRd+UoTnvWk2qOYegpHehTij8mjpA1+zP1mNN mH4LX3b+mdXUCt0B1BP1XjGFG0Ujq2TqVwW592mQiK13W4c47JDevYO88QrgFaMOmNqZWE z5EnRBTvRDcZeZKmcudc0fcUJlTXq88= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=QgsvB1oN/cnvlW92dNAD5N+075D0L1/Jt0SsBfnWtRI=; b=TUtKBu96fL6cDaIJjMTPFnnqLn LHj7U+ypDA2HdpRiO8J2L6d+lOYUekHy0GPF6y6YOJMYi50/kelJ+Wu13FekUJitHJl/i7YLNX2BR 2OBWq+T1sDYZu+eVK2qacKAA+goAlalruABUqrTQC0YjlQAb4GO6A58bs7ObirjFfByPPtKi1osVU wSPbSh9B1anf9oN0zSHU+C+G3tDOyEOyLBMYBoQQfBzeo+KlUpj0pYyGx7YWyIsv9xWlfqPuDTyVi loIa4yhTAKOJjsTzY6RHSB7qocKs3MJHAAQghY4FKU7eVZyJdl2rz8xrhvB9Nrv9F8wG6xlk+l4RW ywky6Cgg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00BkaZ-1D; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 04/22] mm: Convert total_compound_mapcount() to folio_total_mapcount() Date: Sat, 31 Dec 2022 21:45:52 +0000 Message-Id: <20221231214610.2800682-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 3FC1814000D X-Stat-Signature: uz6oie7prckscx1aa5ws8gyxayfwmjf8 X-HE-Tag: 1672523194-653342 X-HE-Meta: U2FsdGVkX1+44BxTXoeRz2w27m4KBSGKEXLgVS/JHdNK5q31H9HD+eZtJvssZLtu8eBCu/RQl166ssxTo+C0w1SggwJAqd/HnuCMPW12VvlYOOHLNatcIJysUoT1jzDWwk1UzATBONTqFkNjtTfUaRI2mEjD09NftTHBv1lMztuUJ4orP+GQKsoPDutRb3GFIOgB2jQD50kYUbNTSOjQSAI5prrEmhXpSvvRNY/ed6tRcPt2ddEqMADmCMcv5/k6yj93FqG+H87Cnijtl93VTBSGovBtPXbLNiDcx8NSI89HA36ok+DXrRH3em/NGfZuzwI+Hw33eNv4yimFsQZ+6jAZKAqFWKNMUV7p6fIxaXFHkw8LrFDYfDan49Iqt2EEFzdoOih0tYyl+8+Syp9HJePjins3zsLOQS39zF4QyiMddpEq53d0LTWTyyV/qr3kAjjSnOpGKeHOWORgoUAAqAoJhuODQ0eJTvwiIvNNzj2rrXUgJDUrTdn65a12mXvbn5Vz86K2qkxaUv3Vn6m6WaYyJ2R0ny9obc/Y/DxmSPqU6WQPZF3SqtKWlfiyzaF7XMWgDCaOoCoFr0PMFxs4XfjBYuhR/pwaS4oLQn8t/nU4nhiYGy02GihEWLD150RhnIP9hwY8Mo6ApKK0bLifJPUM5SXVcbYopEFhltx1M7ETpWFkvyt6pazQih5irOOQNEzJmvA+nYwJ38RzkScBoGrJyYnRscuLYY/sMszcOmEJIV9AahbHozS678ZZ1EO3MyYkMOf5zMw52RRCoQ2aM1ctSYNHpIn1XFjZBQJ1tAsKn4ru4jlqpkoZJy3+Oo7RWnJ6MsJBJKiTQMMTbqXEnfSbzot/JYE75j6cl/t0wgCIoMw57IVviG2b2UD6tiDvBEbhi2OdwzUBiRPFVmTHHDcTyqeNidW/c8Ed+27JOj9Wjd/LdVgLWQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of enforcing that the argument must be a head page by naming, enforce it with the compiler by making it a folio. Also rename the counter in struct folio from _compound_mapcount to _entire_mapcount. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 6 +++--- include/linux/mm_types.h | 6 +++--- mm/rmap.c | 21 ++++++++++----------- 3 files changed, 16 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 7ee1938278f5..0079f779da47 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -866,7 +866,7 @@ static inline int page_mapcount(struct page *page) return head_compound_mapcount(page) + mapcount; } -int total_compound_mapcount(struct page *head); +int folio_total_mapcount(struct folio *folio); /** * folio_mapcount() - Calculate the number of mappings of this folio. @@ -883,14 +883,14 @@ static inline int folio_mapcount(struct folio *folio) { if (likely(!folio_test_large(folio))) return atomic_read(&folio->_mapcount) + 1; - return total_compound_mapcount(&folio->page); + return folio_total_mapcount(folio); } static inline int total_mapcount(struct page *page) { if (likely(!PageCompound(page))) return atomic_read(&page->_mapcount) + 1; - return total_compound_mapcount(compound_head(page)); + return folio_total_mapcount(page_folio(page)); } static inline bool folio_large_is_mapped(struct folio *folio) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index fc44d5bab7b8..15cfb0e24e84 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -306,7 +306,7 @@ static inline struct page *encoded_page_ptr(struct encoded_page *page) * @_head_1: Points to the folio. Do not use. * @_folio_dtor: Which destructor to use for this folio. * @_folio_order: Do not use directly, call folio_order(). - * @_compound_mapcount: Do not use directly, call folio_entire_mapcount(). + * @_entire_mapcount: Do not use directly, call folio_entire_mapcount(). * @_nr_pages_mapped: Do not use directly, call folio_mapcount(). * @_pincount: Do not use directly, call folio_maybe_dma_pinned(). * @_folio_nr_pages: Do not use directly, call folio_nr_pages(). @@ -360,7 +360,7 @@ struct folio { unsigned long _head_1; unsigned char _folio_dtor; unsigned char _folio_order; - atomic_t _compound_mapcount; + atomic_t _entire_mapcount; atomic_t _nr_pages_mapped; atomic_t _pincount; #ifdef CONFIG_64BIT @@ -403,7 +403,7 @@ FOLIO_MATCH(flags, _flags_1); FOLIO_MATCH(compound_head, _head_1); FOLIO_MATCH(compound_dtor, _folio_dtor); FOLIO_MATCH(compound_order, _folio_order); -FOLIO_MATCH(compound_mapcount, _compound_mapcount); +FOLIO_MATCH(compound_mapcount, _entire_mapcount); FOLIO_MATCH(subpages_mapcount, _nr_pages_mapped); FOLIO_MATCH(compound_pincount, _pincount); #ifdef CONFIG_64BIT diff --git a/mm/rmap.c b/mm/rmap.c index 09f4d260a46c..17984eb9f990 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1085,27 +1085,26 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, return page_vma_mkclean_one(&pvmw); } -int total_compound_mapcount(struct page *head) +int folio_total_mapcount(struct folio *folio) { - struct folio *folio = (struct folio *)head; - int mapcount = head_compound_mapcount(head); - int nr_subpages; + int mapcount = folio_entire_mapcount(folio); + int nr_pages; int i; - /* In the common case, avoid the loop when no subpages mapped by PTE */ + /* In the common case, avoid the loop when no pages mapped by PTE */ if (folio_nr_pages_mapped(folio) == 0) return mapcount; /* - * Add all the PTE mappings of those subpages mapped by PTE. - * Limit the loop, knowing that only subpages_mapcount are mapped? + * Add all the PTE mappings of those pages mapped by PTE. + * Limit the loop to folio_nr_pages_mapped()? * Perhaps: given all the raciness, that may be a good or a bad idea. */ - nr_subpages = thp_nr_pages(head); - for (i = 0; i < nr_subpages; i++) - mapcount += atomic_read(&head[i]._mapcount); + nr_pages = folio_nr_pages(folio); + for (i = 0; i < nr_pages; i++) + mapcount += atomic_read(&folio_page(folio, i)->_mapcount); /* But each of those _mapcounts was based on -1 */ - mapcount += nr_subpages; + mapcount += nr_pages; return mapcount; } From patchwork Sat Dec 31 21:45:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086166 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B252C4332F for ; Sat, 31 Dec 2022 21:46:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CAAE8E0002; Sat, 31 Dec 2022 16:46:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 153668E0001; Sat, 31 Dec 2022 16:46:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F36158E0002; Sat, 31 Dec 2022 16:46:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E04658E0001 for ; Sat, 31 Dec 2022 16:46:09 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id AA4F140182 for ; Sat, 31 Dec 2022 21:46:09 +0000 (UTC) X-FDA: 80303934858.10.C543A05 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id 70436C0010 for ; Sat, 31 Dec 2022 21:46:06 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eq9D5ElZ; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523167; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=46PTN6VCfOALWMcndrNvySFnC/1jvl7w4rPJMnjeBsE=; b=UMxQmslH9vJ3/JazVD86N6n7FFgKzgzzKNaEfPiO8JYuz6tGspVbNr2Tk9PosqBl+jlckM GUnEjxynvr7sRUCyae82ZBNrQk3B9RTx9yjQY0jcL5Dy+l05Jn3Kczj1diHIyoLojvkbFm NsCKb8PpypHNqNLu+E0U36l1ibhPWOw= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eq9D5ElZ; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523167; a=rsa-sha256; cv=none; b=qpNvuua1ec4J9C8yCkTJ88R20/eRVxKIvBqRT0yHoson3UGuTbaHHw7eMgQnXQ829lpDNC XMt5X2F8gC9zorQdWLhBe5GaoNXcMC6VhEyvL+LEINK+v83BZLzCIDwZTOpPWDAg9p02fL PAwnG5PS1A8f9N/RKId67Y0H3KGMx1U= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=46PTN6VCfOALWMcndrNvySFnC/1jvl7w4rPJMnjeBsE=; b=eq9D5ElZ4H1Po8lHL8GAgJgZ+D ST/EkWYWOeq9pQozNSuSSy6sjRNM6aapCvgNE502mcom6ebwLNF84rOL+YMYzuJJQuTbUEohbDdc0 3jVS9d6C7B5ctmyIAeZTOMKWr1KLPITPSuT07bGtBx/MqkSxupW3lRQeTfZWQdJiDvCYNXwUMRZK+ cDFAtFrugw2RxRrtPcKpv/9+CQ/AyoUcTrqavzXoye/YRN8NQ6sLEziqHqoYFsxjcKvoZGZgb/I9a gE4HW6dTVEMGdSwYAvBtNyqy8kLTcobokZXcehOq7Hi+dO7whddjtqHqbaKChf5LS0LRU0xcIw/mr w4jFY92Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkab-3O; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 05/22] mm: Convert page_remove_rmap() to use a folio internally Date: Sat, 31 Dec 2022 21:45:53 +0000 Message-Id: <20221231214610.2800682-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: m5bp1rb9j9i5c4zqroznbntkxhzcmhr1 X-Rspam-User: X-Rspamd-Queue-Id: 70436C0010 X-Rspamd-Server: rspam06 X-HE-Tag: 1672523166-84471 X-HE-Meta: U2FsdGVkX1/K5hP+J3uKL8zxv2t8cFeE0rGF4FRjspmY+llGrC/cb3iMCX/YP1xck0DvIxz7g3tIQb2ZKbWkX8qkhjDurgNniKFB7QoyKYL4XZEv9hHW33Zrowizs1dTS2tUP2Pf03ES306RHSLjZsIFK3TZK9ZFl3KYnCzgXYo+9hqyz9mmNsDKYrMLhrQtAI27sxl7k84buNgPXI9+qB3pQXO4FCM38p82mhrn9hD01Tq52ofcuiZn1sY6JfsoD31QP4gu6WnupdgePSGpYkJZPqnwYZdASER0+smpnKtvzTsoKMROqpm8Pt114QL6jRXi/NVoV4eP6MN2qY3KOOyiQoJVZwkv5WRyMiT5do3xKL334nUVU7sN/1iQVhaFLeMUYFcTJDSNlyLSSUkhA5p1LsHv2gL9pFqCk5xEeUyxFj45uh9Rgk6hJX8KuMUCFWy4Bk4cABwTRPIr4mU5RcvkaO+eqCbrMjBGB573tT/Ld+gdPgDWAkprNrP+G9eEtyl+K/vOzXTd+oNJmZMa6hkLLLxR9oKDQzER10wHicP7FsXpsGNQKt2qJMxXIRpHOvm1LXxRJHqowAPp/CQ4pKY0e1wFOmt3t9DElTnv22NaIgUk5C69nQmGynyE5/JUX3cx0ECtzIccHQwcQw9aRa2XgtQpna4f+5Dsd2enjlAXgVif4lqBR1I1/M8zLPRGBnxf8WLEgSnZHD8Gi6/JS7P18xfOHsEzYyQ3OnARp+gYmrLoE+PwpA83O7o8/Cn4JRytvLA1cXtOtdeir91DKSK49a1WCrl1qGyluU51kJbTQxXABA1L/zX90XGasw5rIX8rLhCgzYaHU36SE+k7wuCQhBVdNTilCbNcj/AXsO+GCL+wilzqAUgraxXP00EkkxkpkFfcwkf0+0bJZDOMrSIOT9Ao+j89lV0IvOMCgkJGtEQZyqjD5Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The API for page_remove_rmap() needs to be page-based, because we can remove mappings of pages individually. But inside the function, we want to only call compound_head() once and then use the folio APIs instead of the page APIs that each call compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/rmap.c | 51 ++++++++++++++++++++++++++++----------------------- 1 file changed, 28 insertions(+), 23 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 17984eb9f990..7d51ed49ee9f 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1378,42 +1378,42 @@ void page_add_file_rmap(struct page *page, * * The caller needs to hold the pte lock. */ -void page_remove_rmap(struct page *page, - struct vm_area_struct *vma, bool compound) +void page_remove_rmap(struct page *page, struct vm_area_struct *vma, + bool compound) { - atomic_t *mapped; + struct folio *folio = page_folio(page); + atomic_t *mapped = &folio->_nr_pages_mapped; int nr = 0, nr_pmdmapped = 0; bool last; + enum node_stat_item idx; VM_BUG_ON_PAGE(compound && !PageHead(page), page); /* Hugetlb pages are not counted in NR_*MAPPED */ - if (unlikely(PageHuge(page))) { + if (unlikely(folio_test_hugetlb(folio))) { /* hugetlb pages are always mapped with pmds */ - atomic_dec(compound_mapcount_ptr(page)); + atomic_dec(&folio->_entire_mapcount); return; } - lock_page_memcg(page); + folio_memcg_lock(folio); /* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely(!compound)) { last = atomic_add_negative(-1, &page->_mapcount); nr = last; - if (last && PageCompound(page)) { - mapped = subpages_mapcount_ptr(compound_head(page)); + if (last && folio_test_large(folio)) { nr = atomic_dec_return_relaxed(mapped); nr = (nr < COMPOUND_MAPPED); } - } else if (PageTransHuge(page)) { + } else if (folio_test_large(folio)) { /* That test is redundant: it's for safety or to optimize out */ - last = atomic_add_negative(-1, compound_mapcount_ptr(page)); + last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { - mapped = subpages_mapcount_ptr(page); nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED)) { - nr_pmdmapped = thp_nr_pages(page); + nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of another remove and an add? */ if (unlikely(nr < 0)) @@ -1426,21 +1426,26 @@ void page_remove_rmap(struct page *page, } if (nr_pmdmapped) { - __mod_lruvec_page_state(page, PageAnon(page) ? NR_ANON_THPS : - (PageSwapBacked(page) ? NR_SHMEM_PMDMAPPED : - NR_FILE_PMDMAPPED), -nr_pmdmapped); + if (folio_test_anon(folio)) + idx = NR_ANON_THPS; + else if (folio_test_swapbacked(folio)) + idx = NR_SHMEM_PMDMAPPED; + else + idx = NR_FILE_PMDMAPPED; + __lruvec_stat_mod_folio(folio, idx, -nr_pmdmapped); } if (nr) { - __mod_lruvec_page_state(page, PageAnon(page) ? NR_ANON_MAPPED : - NR_FILE_MAPPED, -nr); + idx = folio_test_anon(folio) ? NR_ANON_MAPPED : NR_FILE_MAPPED; + __lruvec_stat_mod_folio(folio, idx, -nr); + /* - * Queue anon THP for deferred split if at least one small - * page of the compound page is unmapped, but at least one - * small page is still mapped. + * Queue anon THP for deferred split if at least one + * page of the folio is unmapped, but at least one + * page is still mapped. */ - if (PageTransCompound(page) && PageAnon(page)) + if (folio_test_large(folio) && folio_test_anon(folio)) if (!compound || nr < nr_pmdmapped) - deferred_split_huge_page(compound_head(page)); + deferred_split_huge_page(&folio->page); } /* @@ -1451,7 +1456,7 @@ void page_remove_rmap(struct page *page, * and remember that it's only reliable while mapped. */ - unlock_page_memcg(page); + folio_memcg_unlock(folio); munlock_vma_page(page, vma, compound); } From patchwork Sat Dec 31 21:45:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E268EC3DA7A for ; Sat, 31 Dec 2022 21:46:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD8918E000D; Sat, 31 Dec 2022 16:46:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C54288E0001; Sat, 31 Dec 2022 16:46:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BA1D8E000D; Sat, 31 Dec 2022 16:46:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 840558E0001 for ; Sat, 31 Dec 2022 16:46:19 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5CF37140109 for ; Sat, 31 Dec 2022 21:46:19 +0000 (UTC) X-FDA: 80303935278.08.6B2A427 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id E1CFA180003 for ; Sat, 31 Dec 2022 21:46:17 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mBvtosz7; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523178; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wqGEwQQM1+6jn6Zs1tgXOIJhIPrluemv7TOqKQs1G5k=; b=mQv/Pa/KJAirZrKqYC6dLH4mkgN7QylKGEEaofovwjyiD+thcEEl0lQizskfzursVxicoL R61lNEkkrffpqbVJsuggdJPNVws2D7fCr1SWQcfInlkY807vwfx4M+3G59FxJOSyqOXLsC 7TRi6AYs57vBE/ipoJQ0D75n6/qiRTo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=mBvtosz7; dmarc=none; spf=none (imf06.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523178; a=rsa-sha256; cv=none; b=mI4PKZzfeNoCLvLoNUBWCGCl+3PiR/IHue6JY+cD59Dz4dV32ExnZdvro7iUluB721ZvzH eOJbcQ1PnT0tUYGCRfdgyNgHX7CHlFiC4DOTs0JoCZICF7MQHJpCbM6CcB8OwW87Rh0HgB wrs437GV3kGfOL4f42BvH5lQH+l+ZoE= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=wqGEwQQM1+6jn6Zs1tgXOIJhIPrluemv7TOqKQs1G5k=; b=mBvtosz76lFUc9IqSdU2oeQwx7 amuRgReTce38WgCFcMnG+Wb4Klt/l5N1xzC+gbor4fkHRwc1n3+ve1od0ywIqnHUv5iSMiwxfJ8ud AaNLYYXA2pAgVa/82QfUVp7UTJ2d96SDfxtP8ytE6jzxJ1zFyUFgQ8zjh1VtSjLLdDoG6QNv2Rmaq SBleIFA9DVgo7SMO5bQtnSsfxwOty9+VDwNjX6h9vM35hOI96decTMfmdtGJOz9FQSjk8QSzlEPWW VkHxGLJnc8/QU6wiX3CCsV3T2iHdNUi45SDaBHfoowp38q5H6C44ZaqIh+u9AXAAsfHsCeVrT4ybl gnpnHFjw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkad-5q; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 06/22] mm: Convert page_add_anon_rmap() to use a folio internally Date: Sat, 31 Dec 2022 21:45:54 +0000 Message-Id: <20221231214610.2800682-7-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E1CFA180003 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: j9rnj8hjza3omtrjd5aywekuxo9gf4g4 X-HE-Tag: 1672523177-152428 X-HE-Meta: U2FsdGVkX18p2rNO08ohS0MtXioK2ogTXSmJQ63mh6nQc3vo8KUh2e4uYydVWpDA5Rt1Rr0Io6o2l7CZi3vnBoJCEf5XUN7/RMpSd8cZ4/D54AfABERycAm6lGwFNrpm2fXPYMyez8vG0sxTVmqAhO9DpW5/JMjTZbDNUj1g9cej0JvvtBIMWv5Xvuy5BgULRfOQSbDB2Reo3L1DTn+HkKRQIyMClNmko/3zMH0OCp6KYWqMIYYJYAB2WATD329u6jWMW89nhgjjoZZ3KZdcypv5GHf4trk0+dTLxT1X9BJQrXQy0HoixYjEYUJhQo0VNBRqISKVM/w28Hj/iT9Cl/V/E/V7eQ/9CZwUokxgetvc2bejc9yg0WY52sxRn7is/odLYpvZp3gJ7aUw8bpiP90OV36OGyb8NWyjws/+iW1FQUGi45KJdW5VeFbw9ac9riBfjySp0kZacnBXJ4iueRc15U1JM+fE76vOAhJP0mihvUTHxGEVlVFicLooOaKfJMvbShJ/6PVNWjMvdOdPk9bPo0iwGShNlZ6wUe9FFCehCinDB+Br0PmukufEvsy7DEbz2O+LPvyzhBW7puhikd2RLi/nUy+ZioFN3bBSPvk0H364+MsZnpSuwk6q3FVLimY+I3Jd+xJGk2JZmIxSI7MyjlAzlqq65BAv+ofEdM1noNALBqvicht+VpLfn6LXB84F5PahtFM0zOWstsneQlBjTixFyIQ7TWgTtEcj1PUn6AiRemeo0+YlkoLewlv/MR8EpDTDZCcpcEOqBLP2JHstsMhjCtNh1ltR4BY/5RYgA5K6Vpfg3EHTU5K7OkfpL+6+WQ5xAZCfXDW2yRtAi9qo7Htvtnhj5prsvVrqvO8DgWP93Bg0o/0VYO+UAjneM+tlExKS3MrTTwKqY/vW1d4rnfg6aLfoNZ+JG9KXCT2OKxYN3SNcVQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The API for page_add_anon_rmap() needs to be page-based, because we can add mappings of individual pages. But inside the function, we want to only call compound_head() once and then use the folio APIs instead of the page APIs that each call compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/rmap.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 7d51ed49ee9f..a3046a0adfc9 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1214,35 +1214,34 @@ static void __page_check_anon_rmap(struct page *page, * and to ensure that PageAnon is not being upgraded racily to PageKsm * (but PageKsm is never downgraded to PageAnon). */ -void page_add_anon_rmap(struct page *page, - struct vm_area_struct *vma, unsigned long address, rmap_t flags) +void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, + unsigned long address, rmap_t flags) { - atomic_t *mapped; + struct folio *folio = page_folio(page); + atomic_t *mapped = &folio->_nr_pages_mapped; int nr = 0, nr_pmdmapped = 0; bool compound = flags & RMAP_COMPOUND; bool first = true; - if (unlikely(PageKsm(page))) - lock_page_memcg(page); + if (unlikely(folio_test_ksm(folio))) + folio_memcg_lock(folio); /* Is page being mapped by PTE? Is this its first map to be added? */ if (likely(!compound)) { first = atomic_inc_and_test(&page->_mapcount); nr = first; - if (first && PageCompound(page)) { - mapped = subpages_mapcount_ptr(compound_head(page)); + if (first && folio_test_large(folio)) { nr = atomic_inc_return_relaxed(mapped); nr = (nr < COMPOUND_MAPPED); } - } else if (PageTransHuge(page)) { + } else if (folio_test_large(folio)) { /* That test is redundant: it's for safety or to optimize out */ - first = atomic_inc_and_test(compound_mapcount_ptr(page)); + first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { - mapped = subpages_mapcount_ptr(page); nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = thp_nr_pages(page); + nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) @@ -1258,12 +1257,12 @@ void page_add_anon_rmap(struct page *page, VM_BUG_ON_PAGE(!first && PageAnonExclusive(page), page); if (nr_pmdmapped) - __mod_lruvec_page_state(page, NR_ANON_THPS, nr_pmdmapped); + __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr_pmdmapped); if (nr) - __mod_lruvec_page_state(page, NR_ANON_MAPPED, nr); + __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); - if (unlikely(PageKsm(page))) - unlock_page_memcg(page); + if (unlikely(folio_test_ksm(folio))) + folio_memcg_unlock(folio); /* address might be in next vma when migration races vma_adjust */ else if (first) From patchwork Sat Dec 31 21:45:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CDA2C4332F for ; Sat, 31 Dec 2022 21:46:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EEBDC8E0019; Sat, 31 Dec 2022 16:46:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E4EA48E0016; Sat, 31 Dec 2022 16:46:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B730D8E001A; Sat, 31 Dec 2022 16:46:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 820898E0016 for ; Sat, 31 Dec 2022 16:46:39 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5340A1C5B68 for ; Sat, 31 Dec 2022 21:46:38 +0000 (UTC) X-FDA: 80303936076.09.FD7F6F4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id C842940002 for ; Sat, 31 Dec 2022 21:46:36 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oWYqj1gj; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523197; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z6Ypz1vkEVt9rd+JsNwK6A71aPLdMdkfvoLtv8oxYlI=; b=U+4zlJxu9J7O0ZfPpXk0BVLgwn1p3x6Uce/4kVjFnzny0VY/KRgWT84uD6Su9GufAGywWC SsIoTfqAxcu5FSJytNjAprcoVkHL6Tab99Xu7TpXO1/LcN+SFGHoKDBUgtE/ZQKXhTCrUQ YlWT6A8eH4akymBXYrjYeOWp3b9S5wo= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=oWYqj1gj; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523197; a=rsa-sha256; cv=none; b=TlTX+6MwpRKPrUDQGoIHRXNX/XKlXvWbEu1PVwpsTJkE1Di+R7mJKYdMFLCba220TyHpOw GM96QC7LIwSg6+raeJj4gzqxylOiKHpDv3oT8DBEhF/9hwZ+i+wmPvMJTR1VcXMo9yg+wT K8180VLNAjOMGJd5nsbC45sOX2jbfeQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=z6Ypz1vkEVt9rd+JsNwK6A71aPLdMdkfvoLtv8oxYlI=; b=oWYqj1gjRJys+ex1xvHcftLWGX kvibiLxl7Uu0LR5KdIvFN9PljZon4YHCvW4uCsfB7f0nk/mV8taiIChP8T7W3zCEwanGtUVTu82JQ 447bfWkrDtotX3kds8ICoco3GCPegmm2SSdS5LkV2hfSzl21SbchUUFCJfC6gCOC/JVuewBSaTldh 5qkS3Ay7u0RAu20AlZ4czDSMymjK960uyQw3eBwMN/sEM2F3l4JVSWFUK4k1yumI04RYd7W51ov4O VOnSJMrIC/JFGggIJIchSk/ohrnowIaaGJDI0iwVN5YKt+0RYnfTZqQTASIsTFr57gJDOAKo7LjhN NAN76tgw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkaf-8G; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 07/22] mm: Convert page_add_file_rmap() to use a folio internally Date: Sat, 31 Dec 2022 21:45:55 +0000 Message-Id: <20221231214610.2800682-8-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C842940002 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: ipbp8ico7nohhkn387b64ykmf4w6y7r5 X-HE-Tag: 1672523196-261166 X-HE-Meta: U2FsdGVkX18jr4VCSrQ0d+UPA46zYP7mV692a7mUvTkuqRC8qlNYmIsyn9SriNT0ZvUsil1dmaTrMHgCvtHYGo0RmoB8Lr19Z29VsApACGcdGEw7cAc3uuUNVhjKcpe1Y6KS4LooK38ZneNnfLBeR94eUKp5TrpK3RF+b82hoU3U4EXS8kLwN86D4+snS37i1ObEKcLwcXUr6FAFz8mQsohLBB4HYTNm5KoR7bFakzbPe74TNirfdtdkhL3SNcN45rb35wsTbVujUFtKF1wApA8MHp3XuHd7gsNTgzA6iohui9rknF+MtNijV4f733O18iAkppCy9QqUxc0XESxmLWcPWob7GST8no2rk3n1Yov71IrdbouN8pNa+aZEkk5sCGsDIVLkeSgI5UU5EpDqWHIPZRVlzGoL/De/RLqgwKAecebyDEV/OdnAK3BbhCiBRSpQhzRIBdTDotTeSIQnXj21VSxJcX9PWj8DjLNn/0YKEaBQTHueso5X6rhbBU+uH8x5Fc+TKAZbmsbJqkt0EawGMUbL3+aCc43OP+sVrAdnVpdCKtAdur/rFRcskdo0YzHhgBfgJf99HsMzs7vfZDwyk/I4bkp8BosduEupqg0e4b3XRQ7a/0ASMqyWpxaCEhjC6to79uI6rxiGgMX4FOE7sIlzOICoHrCGjJXVNfNTTOizrJK+JVLWQixQLvKyiJliW96HtVRjlCmsIOuS06NrLwxNC0k50eqSYVEY/Q18gEBJ8fDizkGowQOZcZ0mPkuIHceSQC5rb/lK28jvMOpeH5ycCKZkLa6Ss/gzRPdSWD2fHpcgL84/REohLaveOEnjrBQYoi8wop2gakK1ZbheiwTAA2dk8Gl3KDG3y8v6oUcZfFInenmxAjtEBo0BKb4VCMcD1JaZKQoXssHFIs/D6bo5cy0fq0Vo/SyqDCvL3PFPfg9TCQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The API for page_add_file_rmap() needs to be page-based, because we can add mappings of individual pages. But inside the function, we want to only call compound_head() once and then use the folio APIs instead of the page APIs that each call compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/rmap.c | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index a3046a0adfc9..71fbe4a65b93 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1320,34 +1320,33 @@ void page_add_new_anon_rmap(struct page *page, * * The caller needs to hold the pte lock. */ -void page_add_file_rmap(struct page *page, - struct vm_area_struct *vma, bool compound) +void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, + bool compound) { - atomic_t *mapped; + struct folio *folio = page_folio(page); + atomic_t *mapped = &folio->_nr_pages_mapped; int nr = 0, nr_pmdmapped = 0; bool first; VM_BUG_ON_PAGE(compound && !PageTransHuge(page), page); - lock_page_memcg(page); + folio_memcg_lock(folio); /* Is page being mapped by PTE? Is this its first map to be added? */ if (likely(!compound)) { first = atomic_inc_and_test(&page->_mapcount); nr = first; - if (first && PageCompound(page)) { - mapped = subpages_mapcount_ptr(compound_head(page)); + if (first && folio_test_large(folio)) { nr = atomic_inc_return_relaxed(mapped); nr = (nr < COMPOUND_MAPPED); } - } else if (PageTransHuge(page)) { + } else if (folio_test_large(folio)) { /* That test is redundant: it's for safety or to optimize out */ - first = atomic_inc_and_test(compound_mapcount_ptr(page)); + first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { - mapped = subpages_mapcount_ptr(page); nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = thp_nr_pages(page); + nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) @@ -1360,11 +1359,11 @@ void page_add_file_rmap(struct page *page, } if (nr_pmdmapped) - __mod_lruvec_page_state(page, PageSwapBacked(page) ? + __lruvec_stat_mod_folio(folio, folio_test_swapbacked(folio) ? NR_SHMEM_PMDMAPPED : NR_FILE_PMDMAPPED, nr_pmdmapped); if (nr) - __mod_lruvec_page_state(page, NR_FILE_MAPPED, nr); - unlock_page_memcg(page); + __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr); + folio_memcg_unlock(folio); mlock_vma_page(page, vma, compound); } From patchwork Sat Dec 31 21:45:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FBE5C4332F for ; Sat, 31 Dec 2022 21:46:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B75E38E0015; Sat, 31 Dec 2022 16:46:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AF7288E0018; Sat, 31 Dec 2022 16:46:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65A748E0015; Sat, 31 Dec 2022 16:46:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 029848E0001 for ; Sat, 31 Dec 2022 16:46:37 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C667D1205B1 for ; Sat, 31 Dec 2022 21:46:36 +0000 (UTC) X-FDA: 80303935992.25.E7DAAC6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id 4AA73A0009 for ; Sat, 31 Dec 2022 21:46:35 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XxMGaDgY; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523195; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mJzPOwF8dfxyuzqOH4Fd4JOPb4UyLGCk9R35O7JKrIs=; b=VzbeDpjIuCHk0witE2IUJa2cm/AL3qhfXOkpAgaZbWT8DwCZeoAcdEqqINCctEngs7/ynH wENxellPIlm0L43ZEoI1wT+0Qfm6JelTi3ecNKGuBa/BCBYwxNCcTCLVc4NUJZ3mjv8zmV KYWHqzsN8O/cPzroZOwJCkgkErRMfz0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=XxMGaDgY; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523195; a=rsa-sha256; cv=none; b=S/pF00j6o6y9a6amRQfaUe2DnTBDFVswMG4zS8pyQaxzIQspQJVAyFLZsHWSMiZZQWbRMr bM4m/OlBiQUN1mDziDtzl6/ASWCCTk7X07ACL6IKyoam9t/Dz+uKz9w5jozNgKOi9MlCIz qdKyB9DzhTzGbUTQpYN4AKRKh5btUU0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=mJzPOwF8dfxyuzqOH4Fd4JOPb4UyLGCk9R35O7JKrIs=; b=XxMGaDgYDAu/alup8QIF2z4auT ZVUTg5qCD8421AEWXMNvRdArwXcNSOmqQ+DiDakJyZdIRHMtCKH3wDnCPSFuzHWDwSQ7/gAY3elp6 819IlSRl+97lrIC5L01Fokzb25Y55rdPQT5YivpibWmLXSu3lDYwY0RZIT5vVV29NpiJW5Up8cyZ6 SF4hR3nIObGMdtftGEdJbIlsNCenRV/NjFoXAZ/U/N+30HfWHgvwXWNYVR9fL3/IkYvLhJVGyxs4Y 5thUC9uqk4wtpY230blEmWty3/JMOUET75AFdr8isKP/ISSZ77BFw6eniMgdRaj302ag3nv0s8+KU oe0XYJZA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkah-AS; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 08/22] mm: Add folio_add_new_anon_rmap() Date: Sat, 31 Dec 2022 21:45:56 +0000 Message-Id: <20221231214610.2800682-9-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4AA73A0009 X-Stat-Signature: fi5dinwxr8zz7t145rknp8ojiqdrphdz X-HE-Tag: 1672523195-901013 X-HE-Meta: U2FsdGVkX1+b2MmlWQvUcXgC2GYBJojHF4B3s9ZEnREm/KEF8FDmhFgAsy56KT8k0HJxQRclGH6Ai7VwJmInEU4C7cUm7/sWrRdwBJCZQZ7zLS1chgO124eWInqHHbrfh+B25F8I6o5glaw1GE+B44eHwHAGxehfcjO0612CnMzCm2hP230QC0iR3/UudwfnKidFXWn1vE/tcOCJRIvK+1OlUI9xyB3CwnQWMUCNeyxA8xvz/kUQPoySM7sDIAuYQIqUqlLMbf2mSqK1zqt30foYCI0Dzn4q+2YJmqtDMZiO8LhLxy8PfzgBJItx0R45VODVylJCAZEDqVUTGMSBWR2+Qp1WoKgOiDjidQ+NN4TKLzifvaoDOFIL9wbYkjRqNz5EyGpE64UpjUiDsFHSCZZs9bsQdlwRt9hPZne572YfN1oVs9DwP+nXzhtjcNnPyxKGdHTAn5gM9kG4WfJFeucfxW7tHY5vREZeh5DHIY8f60Om+fgAcK7zgUjhYCPSZKxENCevzQ7ODA6DQ6b9sTodHCn1XYLhWMvdqZIueJaReH+gbJdwz3RfuO7RrqSTFeCks8E72aJjJzyaL4oHTIWob80PEXr3G13DDXtNQUIUIQAx/ZbqGcxu492uHMbzbsIhulSt1iYOOVEAsRYl0pfgpIadLQ74kl6Rh/3Vbw4875ae1b3r9pnkCZ337EG3raZgx2rBs2R8UydeOx4y4i4anpgo20Jo6imfoXeAgUkVms+YYs9hs0OlCufyQe+7p3F820sTSFdKA7zyDmb5WV4EV/5UflPBQ6nyPMJBkIecPpsgfbKXX4SsOuliX/HpWde1L5Gf9wS3b9OpZcFcxMsA4BLaCOLNyOiV5IHNbXGS/9jP4Lko2+86+/Jn1n3f0uFcrMEJr3/ENxnY90Le6MwvXiyeC+DvDsfL7IjLWE6nYQImU20WIw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In contrast to other rmap functions, page_add_new_anon_rmap() is always called with a freshly allocated page. That means it can't be called with a tail page. Turn page_add_new_anon_rmap() into folio_add_new_anon_rmap() and add a page_add_new_anon_rmap() wrapper. Callers can be converted individually. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 2 ++ mm/folio-compat.c | 8 ++++++++ mm/rmap.c | 23 +++++++++++------------ 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index bd3504d11b15..aa682a2a93ce 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -194,6 +194,8 @@ void page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); +void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, + unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 69ed25790c68..92f53adc0dd9 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -123,3 +123,11 @@ void putback_lru_page(struct page *page) { folio_putback_lru(page_folio(page)); } + +void page_add_new_anon_rmap(struct page *page, struct vm_area_struct *vma, + unsigned long address) +{ + VM_BUG_ON_PAGE(PageTail(page), page); + + return folio_add_new_anon_rmap((struct folio *)page, vma, address); +} diff --git a/mm/rmap.c b/mm/rmap.c index 71fbe4a65b93..3297b88b5698 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1287,29 +1287,28 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, * This means the inc-and-test can be bypassed. * Page does not have to be locked. */ -void page_add_new_anon_rmap(struct page *page, - struct vm_area_struct *vma, unsigned long address) +void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, + unsigned long address) { int nr; VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); - __SetPageSwapBacked(page); + __folio_set_swapbacked(folio); - if (likely(!PageCompound(page))) { + if (likely(!folio_test_large(folio))) { /* increment count (starts at -1) */ - atomic_set(&page->_mapcount, 0); + atomic_set(&folio->_mapcount, 0); nr = 1; } else { - VM_BUG_ON_PAGE(!PageTransHuge(page), page); /* increment count (starts at -1) */ - atomic_set(compound_mapcount_ptr(page), 0); - atomic_set(subpages_mapcount_ptr(page), COMPOUND_MAPPED); - nr = thp_nr_pages(page); - __mod_lruvec_page_state(page, NR_ANON_THPS, nr); + atomic_set(&folio->_entire_mapcount, 0); + atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); + nr = folio_nr_pages(folio); + __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); } - __mod_lruvec_page_state(page, NR_ANON_MAPPED, nr); - __page_set_anon_rmap(page, vma, address, 1); + __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); + __page_set_anon_rmap(&folio->page, vma, address, 1); } /** From patchwork Sat Dec 31 21:45:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3141C3DA7A for ; Sat, 31 Dec 2022 21:46:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 91D308E0003; Sat, 31 Dec 2022 16:46:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8CD028E0001; Sat, 31 Dec 2022 16:46:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 794738E0003; Sat, 31 Dec 2022 16:46:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6AA558E0001 for ; Sat, 31 Dec 2022 16:46:10 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 36888AA964 for ; Sat, 31 Dec 2022 21:46:10 +0000 (UTC) X-FDA: 80303934900.08.8C64A03 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id BF23920007 for ; Sat, 31 Dec 2022 21:46:08 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rjiLnuPN; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523168; a=rsa-sha256; cv=none; b=pQpuHr9qYRBG4Po52APPsgAQrcbwqzw8iD56lgH0ujVe2tIVNt4g4/H/Coflk2r7impNFz SK5Z+kUEU1lN/ar4dVU+1Y2a9rn6SghHpa4kAahrvgNCL8bf5ueqQkwaGBYV7NB8hZbYXA KdP1o1gWfWKrte7W9Q0WpiK0GbNpOzo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=rjiLnuPN; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523168; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+ZRkO+vnNX+/fzNe9v3qqpyopTcHnBSMjfRq+BW6PaI=; b=2YKkZWNGpCp0xHJaqyjyVm3moXFJjC7hD3wajYNCOrVgtxcFntTFnY2taTI0+pbMIMCRAK XcRu6CqsbtwNpPzx0YRa2/+Vq7s1nc9JYo0b9tYteoZmdz7MvDAYkLjyO0HBfJjsYn4oa2 NHETP241UNMD8mB+jWBLzy7RrNLBSpY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+ZRkO+vnNX+/fzNe9v3qqpyopTcHnBSMjfRq+BW6PaI=; b=rjiLnuPNrBSsoxS/oc5v2FwAFk QbpqRk+baL7uMEgY5jNXwXqdCbIxIexKBuUeC1d3vyoYhpN+XqulemeWaCT6IvhrnEJj73DUqYmZE hA5pgzLo1X6I5p8+dFJZjQzYfH34eVBjzwwCkU9laHykwNpSAyOXZopDs2lVEDcOCh7Mw5TXWxmZZ shEpfROffr1QOBwkFxe2A1A2dDPiB/yGh/nd7wdTvTowduVGgrVotBwI02UnTquGQ8q/7/ZE4iMQ+ CiE78NKruOcPLVyrU6Kl6c/3D9rUHoyUmDCSE3ir1uUt8rJmimCRL9z70GL4qx9N12i0ZyMVWnl9M k1XIAd+A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkaj-D8; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 09/22] page_alloc: Use folio fields directly Date: Sat, 31 Dec 2022 21:45:57 +0000 Message-Id: <20221231214610.2800682-10-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: BF23920007 X-Rspamd-Server: rspam01 X-Stat-Signature: bay3qqmpcz94kbdfmx13uh8zkamynuya X-HE-Tag: 1672523168-46314 X-HE-Meta: U2FsdGVkX19663uiKBoQ2REl9q7p69K+YSFPjlRXO6m5sWL7eTmUKDZLbD6F0e97fbErmc1YR/gxKRc0KJNNHT4vK7UIdIj2YD/rUoN+CXXLYHZojjdJwd0AuZsUXefYJdd4iuC4O5Pr1x8kQN45w+Smd/dMXqQshYrlrujKLdaHTATiM3u/5ptDXKbp4CRiAkTA/pfA03tXN679JXNv4bZXG0GlHHatCdV43KJDd0MT623Uc2VflABSZd5bpY894/fEn33s9rcmBQAynif7bFzkJ3b7Bxd43rnETkjUHX8sSYEehKrkGtTQxr2D9bqDVFwqjTJi9eJZJQHpkDqIPIc6hUG50TqHEfez2p0TtudeT4EnCezF3GlG0YVx1U4uwA5QWVIE3MbUgKvt6MvlDlzl9By7PFEvi8AKIMkXpG8gFF+GMXhSHObPOHMkfnuIKi/6xFndbqJeMQs5Ch/CQ6AtOjVzXyywrArjC0DEG2BT/YS/QLdupYWlUuvwJWx3p7l3e8oQ2Mhjlay20ImEJdVrZQ12cosZ7t47/grX/ZnntghtmTLR+UI6IqKSx167GQUcU+vFXdq3+2y4QvtGdu6XIOZxlwke8dZQklWF8befjecYFwa30bneW9aV3TITRgmYxDJYOb6G0PEknDih2wTD5CmHJRYNXQVrGp8ioJxSLuaDNPwtqEOvzPy7qe5e341THqWa6+JPdZu4m9odWVd+ntbaJpNI7ITMvN4j4cEphNHD8NG7rEUAnMhztvlcX0bJuFV37x7P4nSA/IGRsvFLIpEDlyz1Zf0YEoy/hADl3oZwNvoDKKq8qHYUnjsDHzB68WhDBPEcF711t3vgE46op9GYno2c0onn0x9eBX/B6CZDKbPZLNuj9/0HfDkjBlaFZWAy+j+sj3FvG+mziP1ljsapC8+eRnXRFJ734JbAA6nViq/0gA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rmove the uses of compound_mapcount_ptr(), head_compound_mapcount() and subpages_mapcount_ptr() Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a04ed7f72b36..d309c48afc52 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -779,8 +779,8 @@ static void prep_compound_head(struct page *page, unsigned int order) set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); set_compound_order(page, order); - atomic_set(compound_mapcount_ptr(page), -1); - atomic_set(subpages_mapcount_ptr(page), 0); + atomic_set(&folio->_entire_mapcount, -1); + atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); } @@ -1309,12 +1309,12 @@ static int free_tail_pages_check(struct page *head_page, struct page *page) switch (page - head_page) { case 1: /* the first tail page: these may be in place of ->mapping */ - if (unlikely(head_compound_mapcount(head_page))) { - bad_page(page, "nonzero compound_mapcount"); + if (unlikely(folio_entire_mapcount(folio))) { + bad_page(page, "nonzero entire_mapcount"); goto out; } - if (unlikely(atomic_read(subpages_mapcount_ptr(head_page)))) { - bad_page(page, "nonzero subpages_mapcount"); + if (unlikely(atomic_read(&folio->_nr_pages_mapped))) { + bad_page(page, "nonzero nr_pages_mapped"); goto out; } if (unlikely(atomic_read(&folio->_pincount))) { From patchwork Sat Dec 31 21:45:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75E18C4167B for ; Sat, 31 Dec 2022 21:46:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26B018E0005; Sat, 31 Dec 2022 16:46:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EF6C8E0001; Sat, 31 Dec 2022 16:46:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09B938E0006; Sat, 31 Dec 2022 16:46:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E21F28E0001 for ; Sat, 31 Dec 2022 16:46:10 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B5B20120308 for ; Sat, 31 Dec 2022 21:46:10 +0000 (UTC) X-FDA: 80303934900.05.46A8389 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP id 4327780009 for ; Sat, 31 Dec 2022 21:46:08 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Jwv9mI0X; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523169; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yBZlY4uyFwVGuYu5A8b/2I8OZtf7kPM6YGWYSUCHHcg=; b=7pcj/Rq2HZziCNAtutG7iuB93jI9GKTCVInyZBLDNw793haDVjQye0BOWFLzFChmgE+wfi A2r+0aeOD4yurV3w7mTRPEIDdBbTxAWIzA0zA+x5Fq0BVvSNldzv0F34loe9tgriQzvgAi VD7ZDMIMZw03dP7m4tuQi1a8TxOtxbw= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Jwv9mI0X; spf=none (imf02.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523169; a=rsa-sha256; cv=none; b=RKEVpGkXylBgDvv44wF/l4z1t44OkgJgHXPWNUSQx2gMEwTOooqqJlJ766nQ7qleMGSbzs 6j8qId2/OX4ZhcHKpEAbJ+JeeeIgDuXNAj6UmRGcwiPJqK8dIDAWxSoBQ5ti2hV585zNvR Jd/bBEzibxFAF0CsPybNwVWmKo5YAG4= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=yBZlY4uyFwVGuYu5A8b/2I8OZtf7kPM6YGWYSUCHHcg=; b=Jwv9mI0X4ElTQsfgEYMuhU1ZQX yKA9ZdzeI2GbO/JFKOjEOaQUtLg2kRnVGA+aX8LaTXHcrDyGSHDW15XeQ836Ox+WU3LsFkXz6ja6W Nq55Qtc/DuBmN1HX7lj/dToEo0SxjPm6DicX/tNFmVZ18kfDoqccnMhyYzQ9T5sRuaO6RJgGsz0NW Hk9n0w+yvf04AGGhkFGMiQnD4+gV39YRjlRFL5ORxx3H9z+j5hH8Txl1wEKeuxMmQ/aMdEDv8vYm+ l0fGP0b/XyjLAsgTeFm+245SP2lttUUuiBhAxJS1DYLaGgIIjlFG+WFM0LWAuO0xiIUKM0wmxMK0I jBfxwUEQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkal-Fr; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 10/22] mm: Use a folio in hugepage_add_anon_rmap() and hugepage_add_new_anon_rmap() Date: Sat, 31 Dec 2022 21:45:58 +0000 Message-Id: <20221231214610.2800682-11-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4327780009 X-Rspam-User: X-Stat-Signature: aux3hy9pzjm4m6itr1e1xa8picg1b44x X-HE-Tag: 1672523168-268502 X-HE-Meta: U2FsdGVkX19cVEa0c4DKuXzD8HhcXBMePUus/oqRjXUQqorhhD/0ffuOn41bNS8v1uUd1HAe8x/HsaYEhmwl1RlUSvb2a2wjSIoF3Nole7efd1R31CXr+on/bpsO14eNEolttpwTPd5Ngr12SAVzJlnbSAodiZiIEe7T8oW1+NxX/L7+UJPYZYilPRztaSIZp80BL1PsXjUeLLYIi6VyIlRTsSUqVmZ3XhkkcZQjLPUSLp97kciCtspbUj2QX45VtELVP53XZGZGjtMAd9gW62oPxo36ADwGIz78IiAA48ESPDP8nTh9yJpDIHKoMpuqdL1dv/3GUkJ83vsbYFdzU79HnMIamtv2lVbPoxNfitBu3d//Reb4I+iUScWFZ2WRs20USsaz7ZsjG9io0VXA7PBKleEAxTCrKRyR5rmvEACps5D/FaJICBE8JGMaky1PGg2HTg72FnpuhF1Q1xuGHHtdU7ppaN4F4KVPzZjyP0DY/P1a4FNZwnLSXjoj+oD20amKmNAkQn0V+eVByruygvvHqFNR6/AZ45LXQBzRbfTWPhhG+7E7XNQ8eFPE/E9+PY51aGBzuOkCBPzLOb96TuopKWu1A1wSwNFFnNtiLJtBE3WHhhYDdW1bBGncxZnX/I/Q/r7M7arQu4zq2GE9dc86Ts2RDFbI0YvyJYDcTDfAnjQssNleRis5VlbrTXdkEeFsj/54O6eU3dAR/z3IsnJMd0zVgG4MskGgNiRR+mCeWIiHaGyz7m6+3HqKVslLtbuOg1p3fNW737BTlH29bhlISgWQThngcTYoTP/25fLQyshGeaFwfBw6+1oN8+RKRMe0iOZ7BDNVQu1aD6c7W0Dbp2mXxKFAUzTiE+0HHLzk+bxDERRXpmxsAcGcz4bbYLf2ZDK3oSXjz7eETQBZ8oEnjuhRm+bbzPSngxyAF+v8xiIywXn3oA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove uses of compound_mapcount_ptr() Signed-off-by: Matthew Wilcox (Oracle) --- mm/rmap.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 3297b88b5698..d658f4316bcd 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2545,13 +2545,14 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) void hugepage_add_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { + struct folio *folio = page_folio(page); struct anon_vma *anon_vma = vma->anon_vma; int first; - BUG_ON(!PageLocked(page)); + BUG_ON(!folio_test_locked(folio)); BUG_ON(!anon_vma); /* address might be in next vma when migration races vma_adjust */ - first = atomic_inc_and_test(compound_mapcount_ptr(page)); + first = atomic_inc_and_test(&folio->_entire_mapcount); VM_BUG_ON_PAGE(!first && (flags & RMAP_EXCLUSIVE), page); VM_BUG_ON_PAGE(!first && PageAnonExclusive(page), page); if (first) @@ -2562,10 +2563,12 @@ void hugepage_add_anon_rmap(struct page *page, struct vm_area_struct *vma, void hugepage_add_new_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address) { + struct folio *folio = page_folio(page); + BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ - atomic_set(compound_mapcount_ptr(page), 0); - ClearHPageRestoreReserve(page); + atomic_set(&folio->_entire_mapcount, 0); + folio_clear_hugetlb_restore_reserve(folio); __page_set_anon_rmap(page, vma, address, 1); } #endif /* CONFIG_HUGETLB_PAGE */ From patchwork Sat Dec 31 21:45:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68728C4332F for ; Sat, 31 Dec 2022 21:46:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7488F8E0006; Sat, 31 Dec 2022 16:46:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6AB5C8E0001; Sat, 31 Dec 2022 16:46:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26DAB8E0006; Sat, 31 Dec 2022 16:46:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0489D8E0005 for ; Sat, 31 Dec 2022 16:46:11 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CBEEB140169 for ; Sat, 31 Dec 2022 21:46:10 +0000 (UTC) X-FDA: 80303934900.02.5D91642 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id 4FD3F140015 for ; Sat, 31 Dec 2022 21:46:09 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DDkll6DJ; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523169; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kmo9D0eOQK4gl5x/N93BIarcQ1MlfQ8S0IUSdbRs2jA=; b=c31tP4Y0AH8iwwqThHkG/zJTW6Iz9n9RcrkW0F0aR9iiabhaZPxtTCzSLehNApp+geRJRG cMKUHQ0tqPpoxsZJkOeN3swl5ldKIaOz/SBsOmFsVre/Y2lc0x5sDTsvCCx3Djayix2pKL T+uTs6JTKlgNyPN/0o1e1uJ/5azkIG8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DDkll6DJ; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523169; a=rsa-sha256; cv=none; b=rGhHSMjsIki0Nrx6fkiQWBO+FITtQaIraIdkqEJFbtI7oi8k+Jp0pZJz6jum5wDc7qzugw qQbp7y7HQGze+bETJ1AicNA8mzooR0gF8gghbZXoFXiCYAWJYrVDiDo2c/5NZRYSRp1gu3 BE80ZWyM5i3thckdzw9M0ptU/t0PIqQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=kmo9D0eOQK4gl5x/N93BIarcQ1MlfQ8S0IUSdbRs2jA=; b=DDkll6DJY+yykoXeI0YwdZcMG1 MAoXUCFCO1gxkS8KcXW5DlV3bbXjn/rprAWIV/ypKX3VLyL+6W9JiPPWtEkmgfUxckb5k8ndFncem Wd9x+8OkgcUjApHBY3cpbZB5WPVnk0lWLI+qY9RfEwKSL+F4TBU58AwgdnIPoCwIJuvLCetDjskP9 TzYl4pm9ZBM2zI/yuIl/whZt0mf4x3fp/RhSQQVWc8JCl3XkLNMuknycaFbbjXpwxtoD0vFhFL2If RF6F1DzkREjIWxeW3hrM/oEGmsxUm1Zz+I7J53fYPmXhjIP8a16PHu/5e6Rpo0qyq0/DKeJ1bZbcT bcECkiQQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkan-IX; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 11/22] mm: Use entire_mapcount in __page_dup_rmap() Date: Sat, 31 Dec 2022 21:45:59 +0000 Message-Id: <20221231214610.2800682-12-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4FD3F140015 X-Stat-Signature: ta1qtopqjcwoaot59o6njqta478we354 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1672523169-146154 X-HE-Meta: U2FsdGVkX1+xbafmJ8iGUr7WPegAE0URf3QM1RRDgcR0NX4jZTcbzLVvepkgJG2osDE6tw1T9RZNOCdZfSGUMZzFCbsChAIp7yEe1XeDatOI94k64Yl+utnaDsAiqZh2s/FnWMGxbt+pNgTDjylouybzE931p4UVtkaiJQRKozk8Q0BYcDbET4gx+xQeDI9MZxly7C6J0h9tqDlCwd3R8hCx/v3zc074/HT/QiZZ0uCsLm/uXnd7hWgcRXoaM9HqSBuYEIdFCRNoHhYwjjTOg20WlNIxucBZj7Twi6GIH8AA7erdqL4ZbY1uogTzOCH1qfqigiuJphLlLT5YbgvslumFzHebBYv2xAjKl9JCTs0bz/VZvjQmk62yi6vJH/E7xwqd8JcS4j4MD+7cgLOF/3DtU2QUmzrWMX/KqI5vDRQ0Se1TAuFiVgOguquXiS5bj6eskq8S7UReM72aGeNLYG5Nfu+8SdwumD05lwQfgZulPLVMSTgmyGr382zZiDRInAeug9v2zJDqGO4fkJNzpizcAO4gOTVRSICo6etBcdfbda/iLP5DGM5ZBSPINMiihB+NRsw9wTmbXaF3sbEDhXWsV7g/ZMVqloROV7rMQZYI8vN+Yp9Cmtwsq5VvW/i5LFf6WZG/Fx3loJmz9tUJk3m5tm7fVtDEbDF+UHnGg19L6s5UkJg/UlRBZtuvXL7e78PFYCJ0UAYJ0bhkKbH0wReTM/VFDAc6L2QCUGBVxLgl2U7/7cAo0ffw09sdW5K8aWDDFxg5TmiWd9zuf8ROCgm5bcbehG4t1RUYRuo5o8CtMtuqJMRP5pVey1yAj5SgvLe9ee/RMuzlXoBJlKtmPYKsEnAiGDbRiPRqEwqrA5/xVcEqCG0MQ0+PkbJZ2GoP5t7dcSHlDzD4Vr8eyo6n1F/LP3qDxrulUEhyCMlnwbae1LzLvp3Bjw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove the use of the compound_mapcount_ptr() wrapper, and add an assertion that we're not passing a tail page if we're duplicating a PMD. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index aa682a2a93ce..a6bd1f0a183d 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -208,7 +208,14 @@ void hugepage_add_new_anon_rmap(struct page *, struct vm_area_struct *, static inline void __page_dup_rmap(struct page *page, bool compound) { - atomic_inc(compound ? compound_mapcount_ptr(page) : &page->_mapcount); + if (compound) { + struct folio *folio = (struct folio *)page; + + VM_BUG_ON_PAGE(compound && !PageHead(page), page); + atomic_inc(&folio->_entire_mapcount); + } else { + atomic_inc(&page->_mapcount); + } } static inline void page_dup_file_rmap(struct page *page, bool compound) From patchwork Sat Dec 31 21:46:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086171 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27304C4332F for ; Sat, 31 Dec 2022 21:46:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E0B38E0008; Sat, 31 Dec 2022 16:46:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 041DC8E0001; Sat, 31 Dec 2022 16:46:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD6098E0008; Sat, 31 Dec 2022 16:46:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C53128E0001 for ; Sat, 31 Dec 2022 16:46:12 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9BF3D40182 for ; Sat, 31 Dec 2022 21:46:12 +0000 (UTC) X-FDA: 80303934984.18.DFAF2D1 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id A98E120005 for ; Sat, 31 Dec 2022 21:46:10 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=muQzFTka; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523170; a=rsa-sha256; cv=none; b=3pjvomYdFKyqW5M9r0MytEUxe4iMvzn19uDJ2Y8iGCs2AEN/NYVv1b/OmgpOhMRR203Pli XCiNMWtVo2JiSgrYXNbAQObqIqBlx0ffhhNcrYg/ElfXdQzxVK4m7z2VQDE1gSgeGFzlYY jYh37sinP1bV9/RXhcsJMQ79Pi3i7yg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=muQzFTka; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523170; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oBSy+RoKw5Qche5S64mF1NtwyV9/tXVwrjS1i8/b8Xc=; b=HGb56aXsdrOA76ITvwKS00sWja0sN/AOElOA9jUou4hvoH2eXw5quP4LEH5gvynen8OdNw JXc+g5TzW6LvMAvKYlLedie3HRZOyXWLdAAq9Br31+T1ZMtMmtMuhay4uf+GLv1Eu6ulUF 2s2kD3VVL54YlgE4aUAIdAV/5+6O+2s= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=oBSy+RoKw5Qche5S64mF1NtwyV9/tXVwrjS1i8/b8Xc=; b=muQzFTkaLcGChZJZtYKkOQML1r 3Ob8xiKiGOGlEk1mPc6LoYvNSDev8Y7QphvM8DSgfu/MkmyxafvswtmYrwprQqtZx4mpstEatssSz tSxuvUPTvp2pyEv0Ug8INuQKzpKLgAdcG9/jj5p8WBLkme4z46n3gV6XLbcPllI+gL9vMCoqNK/0O GM8DjrrjDX4GeZq+ES2AuqSQ0oXxr7+VQqBotnUhy5afe/UNndncOsKIKU8iS1tVIjCX6/R88ENwR hPT9qRk/LCiRCAQ5mP5aV/Cli963pjjb7nlQenusPdyaQw+FomWpss5tymTRmkFRNsCI92QVtbPoS aUm4mC4Q==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkap-LF; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 12/22] mm/debug: Remove call to head_compound_mapcount() Date: Sat, 31 Dec 2022 21:46:00 +0000 Message-Id: <20221231214610.2800682-13-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: A98E120005 X-Rspamd-Server: rspam01 X-Stat-Signature: efeiwk8hiysaukset7twq7bajdk4y45c X-HE-Tag: 1672523170-854351 X-HE-Meta: U2FsdGVkX194sBnBO7BZ7IEKRWe+FZ4w6mTLsOWK1BE4BOiyPUW6hXw35NhSWyp9J4Vj43E36joKNijoLh4pDneSX3bR3lxKZY4J5OLLYiCVMGcGpt5HosCBg0aSh+ekOlAhypjeKx7dCYq5QnmV3xoPaxAJH9qbxMuI0lAOKI5SnA2i+1v6IDsXzM5/56CG/AYIt7fByLCVg5rKMtdCR+cz81xZzAWCTyWhAwQPxqoLXX9Ln2QQKwbP0oyFnJEcSXQ+HBWIP1JLR9L+o4DLirpAD+4sWNA9w6paCYRDVNKsxO7aQ+AqZNFiGjleO1CjqdZ60+JmTHjMYlnGUMRMG1dsNbmwSupbT6bcF+W0sZ6ugbtDr+mMpP7w3diGzgxSAl7DGXWinxOcgVSoYY3esLlyjaHLOzYRlN2UR2bfYQHgoSGuzMctda8srl/bXDFSpbqSQWlYMB+ck28f8vLhH+PeOw7+U8/HlDWe2vGm4WvEx6uBQJFF3YxleCZwxzmhe2r91DXsUqXwG53cwGkCG1eGv4ZKtwfJ+kcf5czrnrA1e9qKXXnIes0wwV3zfTyHoQO1NMceA4Mu0Lf7E2IuZwxbNAUw1gxJqT6MAXEgbP8c7TpTpO1XPuH6vfwKOM3TJK6XP6l6hM0TRV2DTGPzhmwm+CiS2hVK/VJGo5RdzkDVruL1ryEua/cIX9JMQC5OuJWKDG95EKU9XGtHpcojecLsXyGLzN+aMpJZRefpuHQtaqz2+PLVbDQP43PY/4+WgFO2kOlsGEiRcXomhrcpbVmLTQz4zS9urFm58OjMJMP+hC7opSq0L0ssT00B0v6gAJ6Th3BomzJSJPImyYcGBveMRTooE/UQ7p+Uz0nJQDRwwMVLa0RcLQCGjVWgd+CcgUe6TVPXL35QeeagOg2Ncya3g5Ka/ttx+WF2HzohHChuD6cu0t+q8w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Call folio_entire_mapcount() instead. Signed-off-by: Matthew Wilcox (Oracle) --- mm/debug.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/debug.c b/mm/debug.c index 8e58e8dab0b2..9d3d893dc7f4 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -94,9 +94,9 @@ static void __dump_page(struct page *page) page, page_ref_count(head), mapcount, mapping, page_to_pgoff(page), page_to_pfn(page)); if (compound) { - pr_warn("head:%p order:%u compound_mapcount:%d nr_pages_mapped:%d pincount:%d\n", + pr_warn("head:%p order:%u entire_mapcount:%d nr_pages_mapped:%d pincount:%d\n", head, compound_order(head), - head_compound_mapcount(head), + folio_entire_mapcount(folio), folio_nr_pages_mapped(folio), atomic_read(&folio->_pincount)); } From patchwork Sat Dec 31 21:46:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 686CDC4332F for ; Sat, 31 Dec 2022 21:46:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01BF98E0012; Sat, 31 Dec 2022 16:46:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F0E298E0001; Sat, 31 Dec 2022 16:46:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB0568E0012; Sat, 31 Dec 2022 16:46:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C784D8E0001 for ; Sat, 31 Dec 2022 16:46:33 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A37938017C for ; Sat, 31 Dec 2022 21:46:33 +0000 (UTC) X-FDA: 80303935866.16.14D45FF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id 3F644140002 for ; Sat, 31 Dec 2022 21:46:32 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=m4F0hq9k; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523192; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X4TNv1NRkww3z439/1Ja0BlYARXRlvPuYO9UFVVk//A=; b=X4p9WL/pNoQe+DVFCRr8CmxT8DfvvMnZtYT4W59dYiMunaDqPPiO/XLkAuA1wvqT374tJg ZP/8QSBfc7pNAFcZTp0GAH1VQyqnez8OnNLJ9ssbyyhcLyCMJ2JRXzNPNJ3BneiJNd6aWy /rQe1pY5I8mTzn0PV9fIXiY9cAjfRFo= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=m4F0hq9k; spf=none (imf26.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523192; a=rsa-sha256; cv=none; b=duDWkS92QkysxnvhD+5xy7vfmYuTulQIvEBMkvu5xFZiqj3BsKfx80XzZM1sPUgtZDOFhZ CsMHURKP7KBFNiIiAkxKsM81GtQIr9Oqk9hsks2jEiHNq3mrw9pfbTWSDgFxuLXf4b2MSD 24ZhzGYKhk9M7RtL1R8WHj1fJ1qnLdM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=X4TNv1NRkww3z439/1Ja0BlYARXRlvPuYO9UFVVk//A=; b=m4F0hq9kyTXF/CzUVQKaVuQdBI dqveLy0dLteDjV0UpIKK1MD+DbI1zI5E3m2rUPW6aoQZ4MbhULN66Bw2e16DDmGrE0tIfhdO3a10Z GCBnSMmxtbmd1fCkKtVWWyKU/wmLCEY664BEX5trVARiePN1wkuISeD1MVIfvE4nneYAnpxGIm4BD 0wDKhFiyHFq1j9Tn6mHPuGPB36JodVNe4yol6EYkCDuOb1Mgxmw4aN2eQsjVhYY+EYp9l5RWCe2+T RRWpIpLHTbSVzvnkVKe63ge4VleStIqvwBtfCpjlttjVYYEZYLopVYez1RemzMntR6rfOhsqwCBm/ 1HzDUzwQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkar-O0; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 13/22] hugetlb: Remove uses of folio_mapcount_ptr Date: Sat, 31 Dec 2022 21:46:01 +0000 Message-Id: <20221231214610.2800682-14-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3F644140002 X-Rspam-User: X-Stat-Signature: 4a4gm7eytzoqfingytyo6487w5w1kwap X-HE-Tag: 1672523192-657754 X-HE-Meta: U2FsdGVkX1+7sv18HK1Li+vNAWIEINfc8kg9PVBMciMfw8dT5c5BJ33jVVnBVI51SDvpEiIsRL2r+1MGRGaS9j1OqEhuoJ6aj1OsaKGVoMYkYekMwwB1wVYZvloAKgs3g9jlE5y131jJS9BWON28hgdr9uAqute0W8uE1DZIcnB1SJltNYtSv/Wl8D2xKQCDtGaLFAmTydvEeDUe23PG5qJBefTy3DSsQvadUZY5/LpRBXoyXyCNn8g6PLCAFeolyHvl0LB1MVDyZHjgrphv/sVpI3oIgbal4w8pwsjlcibZdm3ek2KVmM3Mr0sHxCr6mh1hfXuHd+/Zw2KJMyDKg8Z/24ST15/q/yoIdhJ9g1eLfRyux9BTdikWxeyYDDvBoOYcIErSuFOicHNU0GzUTIVBnVaeoQW1iJAhT23qvMBiyimefN0RXtlMLysMRIAFu9GcU0Dws8ClWxZ/fz3DoJf8Twndd4FSMOij41MZGeeHXB/gO9fhiqxpydfILNuJf1M1WRGcOjwVxqFK0140gaJjlEvuN8k0Gg1OJSEsG3APDvGDPNzJOP7d2V6PAgjXO1XrGxAwAIr65+J77y4fEAo6r8K7HUhKBiqDzeYoY6n0xVV2E+NOUh6Afh4g5N6kbO3FUFeRzcg24SC1yzshTl4j2OJ3vWGS9gqaVV6y/dZ+bcSSI+/kA6CsOXHLPHth9sCZ9pO/4Q8pH/e47HpSaZp64aVDjLm+vNVBFqH574CEqTKjrFKxftQYbOcP6zMsiuY50mRCM1+X8Whb7k2TeAZJfOOqKVAyI8cRB8a22X2uM+vsCtTtRb7hgYYO1JMSiu1j00GR5e+m5m0K7rpzMzlU8F+nRVBmZpE66N6nrJEPy0sJOAYnjTZF1fZ9rMK08j5co7kG1FC832AiwxPH4blEbUWlkfHjt72WJMX7E5mxZYoRyutp4A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use the entire_mapcount field directly. Signed-off-by: Matthew Wilcox (Oracle) --- mm/hugetlb.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 55e744abb962..0a6bb471c1a7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1478,7 +1478,7 @@ static void __destroy_compound_gigantic_folio(struct folio *folio, int nr_pages = 1 << order; struct page *p; - atomic_set(folio_mapcount_ptr(folio), 0); + atomic_set(&folio->_entire_mapcount, 0); atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); @@ -2000,7 +2000,7 @@ static bool __prep_compound_gigantic_folio(struct folio *folio, if (i != 0) set_compound_head(p, &folio->page); } - atomic_set(folio_mapcount_ptr(folio), -1); + atomic_set(&folio->_entire_mapcount, -1); atomic_set(&folio->_nr_pages_mapped, 0); atomic_set(&folio->_pincount, 0); return true; From patchwork Sat Dec 31 21:46:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086184 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC108C3DA7D for ; Sat, 31 Dec 2022 21:46:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6B0FE8E0001; Sat, 31 Dec 2022 16:46:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 411928E0013; Sat, 31 Dec 2022 16:46:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17C498E0014; Sat, 31 Dec 2022 16:46:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DCDAE8E0014 for ; Sat, 31 Dec 2022 16:46:36 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C129E120486 for ; Sat, 31 Dec 2022 21:46:36 +0000 (UTC) X-FDA: 80303935992.23.D5F9ECD Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id 2F24F40002 for ; Sat, 31 Dec 2022 21:46:34 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hLDkGoc0; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523195; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uSV1JGSnX8tQjtuNAwlCQ8YX4bimlKh/5v7VMgUi/M4=; b=3QT7+RVhqzKRyfd8MZufzr5tdh6GcDD26DiTc96uGL1CH11Eew2nWVkbiJqfjvXOxz202x mzfdy58o3ZyuGMObR9iJOuSzJk/6/um4O8HIMp7g0EHwCPifqyoizYe2r43cbc1EF6IBP+ MJL81eTagi/Wha7d818c+nUwmMmAQ6k= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=hLDkGoc0; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523195; a=rsa-sha256; cv=none; b=frGh/4sILFvBZeMyYyKacd2B2BSH33MjCyopi8X1eOTBSUMO6CX+0WT+M4ZDGKM+ygcNmg rSZg9X2zT1wYQt3e+vezWKNH9KDbnZ0Ej5MrzojXohTLENM3NAqd8KO+S/4HbqGqj584Zl nX9sgQDuLTDuMxWxSlbpcGdYUlVcaek= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=uSV1JGSnX8tQjtuNAwlCQ8YX4bimlKh/5v7VMgUi/M4=; b=hLDkGoc0/5s6uPr/7sD/fKJZB9 Uk4gcU7y0/aFtOhfJKWsngdUL4O5noZbXaPDpzDRPbdpFYoqFuNlmN8rpQyJ/UGtJjzjRz6zXt0Qc FMD0SU/sFdwWV1HsEflczN51lyJKji9ot5Ah5wlbe3XKw7GAqcj1bWVDqLUu4tikntQASCT27ZFH7 3CTnZXSueLwfoszaCVdAYc7mZTUhx2pTtmk1wcn4l5Af8u4fKl3uLcqUxXyeVVGN4fyEyODdKH4u/ mlP+BDZi9vu8vM6oRZa/pNTrbX63W4xXnUDrMSfdceF+zBuhWtBmCPHJ+j9zJGCKJit783RrnsBLI /P+m92Yw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkat-Qi; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 14/22] mm: Convert page_mapcount() to use folio_entire_mapcount() Date: Sat, 31 Dec 2022 21:46:02 +0000 Message-Id: <20221231214610.2800682-15-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2F24F40002 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: ih1r8s45yz8pskrdijxa7gr5aqfzi9wb X-HE-Tag: 1672523194-647145 X-HE-Meta: U2FsdGVkX19alaR9wBfTv8Y8qLoyPLgsAquWJpac4f1YMgkA/tYy+G0ZmMBGv2Id/tXluSTd6iuHBr8MIxl2uzCzysqF6mbedCFALy+M5LrtqvR8kz/YTVCeZZTyT9jbML/Y/41738QyhterDzdyEQaqjv7GDmb6d26FUkKym72jBSAVcJBF/JT1YELH1Dy0Igfw9wmlousLpJVlvzPZZ6EU1urkvZ3wszsgTH/B4/iodDn8ChcAj9Sv8hnspzASryu7iVzD7DcH+P59/exgwQCw9Fm8Cg0enn/2liwtMQacTHHpoVU6nBbuf1jPkbJ174s3R0d2WzLOg0TH60E84mvUvgpLiPXWJUnsfvPfiAbiVBGpdTb4+IFcnsGvtlwP64rJU8z/kXzAfJrjlL1j7dN9CuzWnlklqweV8HH50nka5s6FJE3/BNcL7iuSHEWVyYpXYE0/6TWmY9UpvIgxOPi0lsHrNE3PEdM9RUutvAbgBmW0p6JBfu43+m01BbRMVRg7sVrC4cEhgUgOC0+bZi1H3GsMrSH2XIz34QWUsm2tBEzltNsy4lD1g1h3Lm8SLUMrRD3+0YEMBxW3eAi7TDhogh9YGTEf7bL3kBvH7V5LUt3So08bAFiq/5biy1WxDfKBLjV5Sm8Ou4Yozn/5TeMA+c3zuWjhSKxAD7tpDAOcu4+MKD7Bwi+eSxAtBG+kd74DGt+RBlgbGq0RFFcv6UmHJgGIF0dpS5bcUWCVhUejf9MAs2+82ZXTfrJqSjHsJ6YbZ7qDBnNiqLpZ2YAdgJH6CFap57F3XTcKmZWnBhRgdybnm3KumKoD0ouvBT0LEYVsPZNNcBeufUUZw4B9cTdKAEKrNGLNTizA0+QPRGdV5lkH3C4vQksUXjYDf35CLvSc/qgd7ahn/6YfVT7dkFvbesoKhwlg3TtQDZO5ZuHnm9vW/Y7+gA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Remove a use of head_compound_mapcount(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 0079f779da47..cc6e94647ed8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -848,22 +848,26 @@ static inline void page_mapcount_reset(struct page *page) atomic_set(&(page)->_mapcount, -1); } -/* - * Mapcount of 0-order page; when compound sub-page, includes - * compound_mapcount of compound_head of page. +/** + * page_mapcount() - Number of times this precise page is mapped. + * @page: The page. + * + * The number of times this page is mapped. If this page is part of + * a large folio, it includes the number of times this page is mapped + * as part of that folio. * - * Result is undefined for pages which cannot be mapped into userspace. + * The result is undefined for pages which cannot be mapped into userspace. * For example SLAB or special types of pages. See function page_has_type(). - * They use this place in struct page differently. + * They use this field in struct page differently. */ static inline int page_mapcount(struct page *page) { int mapcount = atomic_read(&page->_mapcount) + 1; - if (likely(!PageCompound(page))) - return mapcount; - page = compound_head(page); - return head_compound_mapcount(page) + mapcount; + if (unlikely(PageCompound(page))) + mapcount += folio_entire_mapcount(page_folio(page)); + + return mapcount; } int folio_total_mapcount(struct folio *folio); From patchwork Sat Dec 31 21:46:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086173 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DB2DC4332F for ; Sat, 31 Dec 2022 21:46:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A31C8E000A; Sat, 31 Dec 2022 16:46:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 329378E0001; Sat, 31 Dec 2022 16:46:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A3588E000A; Sat, 31 Dec 2022 16:46:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 030868E0001 for ; Sat, 31 Dec 2022 16:46:16 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CE7E48023D for ; Sat, 31 Dec 2022 21:46:15 +0000 (UTC) X-FDA: 80303935110.27.A2C13C2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 63DE8100003 for ; Sat, 31 Dec 2022 21:46:14 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C0xHObCN; dmarc=none; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523174; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2xGuPcvz5Lm1GKzxr0YAaEtd00Qs958dmzHJkyKnDCA=; b=Ma6daf2nbXpBlxFXsepYyJVrzkIOj/X5oAfLwSYjvX/DcZnz+luGOXkRRYiH9FcaPrKGnm omz+rL6gWhic2mNJXOgEX8/Vv8he8tpm7Ohp+r73UITMLhMSnim6FPY9U+Q6uW8hmWRMlq g96XXhBZ8mgcYYwRIiBJXOTlzGMXHvw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=C0xHObCN; dmarc=none; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523174; a=rsa-sha256; cv=none; b=Qe3+LqGOM6CDus8X9oI08dENvDfDfiAnXl2jsFx6UKK0bTohDhETEHROgKAg54TBlLKEL6 +TmB1Okv0dweGrqkSGMRdW9AHtHOp8WLM/WShs3XzSJgh34tftPhKn3pLPeba3klheQM01 Kv+BISwcWOWx2GTzKQdbpkFTO1NqoXo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=2xGuPcvz5Lm1GKzxr0YAaEtd00Qs958dmzHJkyKnDCA=; b=C0xHObCNfOd0cvYx7mvzmXcvaR fSQ28WH8biSuZddMRVZ9Sn4gCCcy2bRcA2phlpNvB5FcxVkXNfi5Z3UdkpislMvaQAh8cXViJtV8H MSmOPSynU7GjzSzqsTPzOjZNr+DkQ5pRkAyXLp/HLnPomURUVSEJNqArg4tgkAc79SsE83dRiarR7 oZG73fhzAVACLxI0CzWrW5EZjaK5+m0SM9n3xJRFP7C/o9+OtCWCPw9unXJ3MElLmppD688fuYHXq Q/gAbJGD/26GMxj1XjTGzlHgNCz9Nu4XhSeNWoxrHIYiSyBf+QpIgdabINCFqVkvjfObzxdAex/lO mbMHp7BQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkav-TQ; Sat, 31 Dec 2022 21:46:13 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 15/22] mm: Remove head_compound_mapcount() and _ptr functions Date: Sat, 31 Dec 2022 21:46:03 +0000 Message-Id: <20221231214610.2800682-16-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 63DE8100003 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: o69j8pz9tumazognbfiud7mu1kqc68dq X-HE-Tag: 1672523174-560379 X-HE-Meta: U2FsdGVkX18Fi/i2m1UGyFedvK33rYwZSI2aq5YvxACbmPd4vIWpfkcLxZc9DvuDSZxet3hSAVqXucMl4DSjcFYosulg2vFNnH7IFR+B2/8XXn7MQa66bhTUyZQBYWRzG9kT+JJGWv+xwI0RuO6VK5Anx6f4K6ZlIp1VgX5vi9likmLHkqrrlYgSaeEyqpDPfPHckQFnbxkSRvqpw8O+MQHnzDtcZCm4koQcm+K8XHSBIuv6galovrrSfMFSHQP/d92G1j3MO6TkCFjlAOJPgmR0emO+81g+kXu2lhw/18apR02uhPH+1S76K0h8ECLmAoPYnuNL+gnE/VXox7Sn06K5G5Bu4FoSaZ3D/bZ766yMDaJCx+BPo5K65cm9P2r/31/4KxUvs8Hi5kObgLe3rzsvH95C3EW8Ja5IuzEU+YM6lu6imA/SGPYC7v07oemHxPzCYkd0zMW+IusIUABJR4k/4uQEf9dqVZA7MEq1YeP/Ix7Bgy7GcByJRq4EYPAOpajjWmdW9Buy4MepdtJsHORpMf7k63QyXaxKrcjsXBXtJ9OqHB9qXZ7wDA4+Nmb306CvEuToHJ9MH8oPDRq906otZAbEw447EgMIi44uDDq5DaO+F4ok+wKyawp4x1OdBnS1D1oJnV2M6qcD8aSMx+/g9L+FbokSHPWPiSlAS+sA0j1LCYw3zQWjhgeiN/dfskrm5I/U2LcVSe7ekFGL/iZQL408JG+FNwvllCyLE16h44HEO0+2WcaUNObXc2GRA/efaeJiZ/pPbKGTIsw8OMxtvbms0yo8DznYgOGIJDv0QTy1qWqZ+7J61oxS+SQ7ULSlH2+IJGh+JjGZPGIkBEVcjzdBGSjEErzpgSVuK/oERxM9Pswa0WMm0Zq2QzhLXN9BSt6EF7qsO6s2y7AxdNc3J/d1EnflpwaeZeGXdEdS4kOoIJq6zw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: folio_mapcount_ptr(), compound_mapcount_ptr() and subpages_mapcount_ptr() are all now unused. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 15 +++------------ include/linux/mm_types.h | 16 ---------------- 2 files changed, 3 insertions(+), 28 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index cc6e94647ed8..fca37afa71c8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -826,16 +826,7 @@ static inline int is_vmalloc_or_module_addr(const void *x) static inline int folio_entire_mapcount(struct folio *folio) { VM_BUG_ON_FOLIO(!folio_test_large(folio), folio); - return atomic_read(folio_mapcount_ptr(folio)) + 1; -} - -/* - * Mapcount of compound page as a whole, does not include mapped sub-pages. - * Must be called only on head of compound page. - */ -static inline int head_compound_mapcount(struct page *head) -{ - return atomic_read(compound_mapcount_ptr(head)) + 1; + return atomic_read(&folio->_entire_mapcount) + 1; } /* @@ -900,11 +891,11 @@ static inline int total_mapcount(struct page *page) static inline bool folio_large_is_mapped(struct folio *folio) { /* - * Reading folio_mapcount_ptr() below could be omitted if hugetlb + * Reading _entire_mapcount below could be omitted if hugetlb * participated in incrementing nr_pages_mapped when compound mapped. */ return atomic_read(&folio->_nr_pages_mapped) > 0 || - atomic_read(folio_mapcount_ptr(folio)) >= 0; + atomic_read(&folio->_entire_mapcount) >= 0; } /** diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 15cfb0e24e84..381103c81cdf 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -421,22 +421,6 @@ FOLIO_MATCH(hugetlb_cgroup_rsvd, _hugetlb_cgroup_rsvd); FOLIO_MATCH(hugetlb_hwpoison, _hugetlb_hwpoison); #undef FOLIO_MATCH -static inline atomic_t *folio_mapcount_ptr(struct folio *folio) -{ - struct page *tail = &folio->page + 1; - return &tail->compound_mapcount; -} - -static inline atomic_t *compound_mapcount_ptr(struct page *page) -{ - return &page[1].compound_mapcount; -} - -static inline atomic_t *subpages_mapcount_ptr(struct page *page) -{ - return &page[1].subpages_mapcount; -} - /* * Used for sizing the vmemmap region on some architectures */ From patchwork Sat Dec 31 21:46:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0877AC4332F for ; Sat, 31 Dec 2022 21:46:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E4C68E0010; Sat, 31 Dec 2022 16:46:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 71F058E0001; Sat, 31 Dec 2022 16:46:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 54CF38E0010; Sat, 31 Dec 2022 16:46:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 421A28E0001 for ; Sat, 31 Dec 2022 16:46:28 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 213D51401F0 for ; Sat, 31 Dec 2022 21:46:28 +0000 (UTC) X-FDA: 80303935656.16.6590AA5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 5435E20007 for ; Sat, 31 Dec 2022 21:46:26 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=PK64GSLU; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523186; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0NLqJEVTuj2oTskxw8IoL64ubPWS69O0/7nY9tIXboA=; b=lHtbpxcUlStdR6fQdxHMzTpHlXp8H3XUQpBgo8Zaxz6ZzkXlSvFo9RFsHGwtsT/I0zRjjv ScPJs+60EmVy9cXXOB/Ak1YxxUSx4px+ioav11kaxywqlDdsXnO+gXOKcMlnUKm2OX3QmI ermfj6LhZZLkfVQtlJc2i4jJSjdrbCs= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=PK64GSLU; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523186; a=rsa-sha256; cv=none; b=Ba+8WhwmN+N5MK3uaWpdtDabsuwmGg4l3BYLwDy35KjzBClZLfV8ZORkstWczzO+5+vTP6 NjDKOqWYuyA42eEI+7VXRswppgzhv6y+TO6zo7HCp/CeDoIgPlxlIjOvem55S/pPPsAHfo 4ixP6g5ieYASiRi43iliPqbCpEvuWB8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=0NLqJEVTuj2oTskxw8IoL64ubPWS69O0/7nY9tIXboA=; b=PK64GSLU0AUhEeCdj+/6ow6t/y P+fwnbsvne2dXrs9ilASYoYAJvHzA/fso0Y/n8U1LDhHwHnxaVMzTASrtjApKEr5ZqTDcPbcw5INT 0fbSJmpUKvDNxXQVdb9NZMLrt8dtu1q/0F9F5p9tcoZSzE0c7zOJ2aUCmutOu5Bm4b+r9d22UQCrs yBGyn+4owb42BGu7CA/gt5UJeptl3lkOeshtt67tJswPIxUpirF7TKFdb+gJgo5FivmmegjY7S3Go 3+PgRQBM/rBlOQv2bg4bgw+E+IhmdUH8gJr1uEztugzlMbjqQnN/1bsH9qBMHY4h1DcmTERx5QZrB 7noOmkCw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgD-00Bkax-W4; Sat, 31 Dec 2022 21:46:14 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 16/22] mm: Reimplement compound_order() Date: Sat, 31 Dec 2022 21:46:04 +0000 Message-Id: <20221231214610.2800682-17-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5435E20007 X-Rspam-User: X-Stat-Signature: 9c3x49sbjbnbgtkq3tyrcu7hzwtu418d X-HE-Tag: 1672523186-180117 X-HE-Meta: U2FsdGVkX18NKSjaDjAEcUHDrGWGbIPsAPSwiZ5lVWpQuJiCN8kKiHIiJdeS6xNyUtu54Nj5ILzfRZWEk9sXHZFNkaisXQdHLx0VclNOEqO3Tx5jwh50zJdtJoDpNsIwp1fwDeVAQMHQYv7ALjUKxHXqhInkSmOvQbYi4SYgbAMqlUb/JCnd/B4eAfOPfPEnjzrIza+p02uwNuKWnAlGjUpfTDjKUjLa3F8U1pn5Bny88aobShL3tm16lCfOl7SDpcxKbPGffXuKxpVtmaw/vX6AJWuE/Os7uYRm8K6NDCB2/joDVwBPbsxwHuOYYsUp3hAZHnzKdBzNnhvjsOfpegw6lWPE0VLckxPrvH9Xcv1JdjCvnD8+G3bC7IM3Fc1cIXQ0WO0RCm+82yuoQ6OXFuHbOqY3M8oQONO36kBus6KbPEuNYNctBvho1QJgNcJXxhvyOKt8dLTxwCcDvJwRHJpy3MYHDxYKg0b3+UF+VDPIRhGpKeLSpQV1xQFv/Vwwdgu3+rDkOfj3xyU33XAwn/qU+K+IlmQKOkrghcVq9uFuqHaMiNjxqzeyB+Hazo1b7lJph/Yb852sq4XDgxYPiumMDx6ch1LB0T99vGDVzw+Xsmpu2C3fzNlUz/DClo4hqnnLt8PGWenHpYPmV2miKF2HQIQraU4sV6fj9q8aEWfiiWuC1HQVhk96CjegP6A6dvbIHPy6jlpc0+DQLseXVuLU4AtptP+rpSlCPWhdjdyKLk6c58XJBsOIrGj+qJYE+tISF/vVImEhUNYFSZj8HPB4eeQ1Mhr8YWXYj+BkuVRV1QDlGg1Asg3g3TW971zIqCwFzgpJpOS9rHNs/PcUcWI6Ptkr4/uF5N5jq6Wk1eo7IA0O+V4F5OQgKUJQDrMy8oPU+4eWV/oB04EFJFvPGQGhlFpzMrcTL9n2VpDfmTfniYNu8ArQAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make compound_order() use struct folio. It can't be turned into a wrapper around folio_order() as a page can be turned into a tail page between a check in compound_order() and the assertion in folio_test_large(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index fca37afa71c8..19bdb4b0e00a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -714,11 +714,20 @@ int vma_is_stack_for_current(struct vm_area_struct *vma); struct mmu_gather; struct inode; +/* + * compound_order() can be called without holding a reference, which means + * that niceties like page_folio() don't work. These callers should be + * prepared to handle wild return values. For example, PG_head may be + * set before _folio_order is initialised, or this may be a tail page. + * See compaction.c for some good examples. + */ static inline unsigned int compound_order(struct page *page) { - if (!PageHead(page)) + struct folio *folio = (struct folio *)page; + + if (!test_bit(PG_head, &folio->flags)) return 0; - return page[1].compound_order; + return folio->_folio_order; } /** From patchwork Sat Dec 31 21:46:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85890C3DA7A for ; Sat, 31 Dec 2022 21:46:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DCF88E000F; Sat, 31 Dec 2022 16:46:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F04A8E0001; Sat, 31 Dec 2022 16:46:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9AF58E000F; Sat, 31 Dec 2022 16:46:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CBDBB8E0001 for ; Sat, 31 Dec 2022 16:46:24 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A94A81A05A5 for ; Sat, 31 Dec 2022 21:46:24 +0000 (UTC) X-FDA: 80303935488.08.5E3282C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 2169220005 for ; Sat, 31 Dec 2022 21:46:22 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bI0pe4Vg; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523183; a=rsa-sha256; cv=none; b=SvRTZSXcN6IXkbkBFiO1rRh2mks3BtG8M9tfCa8rkzwAp8MGXPvP6LLIJDGcAMkG/G/hQs FU7DP+ELfwRchhB/6h8Hffw/y0g1Gk8DkCKU3r5Kv8az++vxCOCkug67d5kGjJnOneA6N5 KR0sTbyZYiOVxqUGTyNRUfilj5TIYio= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bI0pe4Vg; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523183; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JanrAscp8O841ylPQVyLGAsbvT0lHRCieI9tsy6zFJ4=; b=G6Z/bG/TDYVsnQHflPPglg+JO/z7XRs4k2K335VRqcJ2xCaIlQy6fnOIZkpMPt+uETg7Yl me40qONBRgMSDZ7sIVl/Urg2d9xK5KHo0cgl+HPeeDg3fS7a/9wjwSm/320/wF/hXNRv8B 9tIqPgvFSXvd7lszibHPoPOPiLD1zLo= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=JanrAscp8O841ylPQVyLGAsbvT0lHRCieI9tsy6zFJ4=; b=bI0pe4VgdNMI7TXhPTmWe2qvYd uHR7EeMrbPJiZBmMr24bdgYy1vup4hrj/x1//HFdb4endlIaQt6pqkAyoM6Qwg8WFvfeGcoT0K6wg OWhJ2Ux/kAA899G9qG+EdxPuytg1D1RCre75HQVa4UdXLyA85qlJPy3jxYAeE6Cy8ElQhDMaDaVWs AkPKjZhQVdTXMtj/0aVsUX4FFHN+DId320c8luzin5Lnxf8gHSIH+KMI0q5jyLYSmNNki3gZhMcsx mW4Hs5JAUo4abUD8tLdZaZONMScIsnePYUGcA3lLWB03b0Pgtjkgns0/Lyz8jA2ntxfuWupYlQRNq xZukq1Rg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgE-00Bkaz-2X; Sat, 31 Dec 2022 21:46:14 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 17/22] mm: Reimplement compound_nr() Date: Sat, 31 Dec 2022 21:46:05 +0000 Message-Id: <20221231214610.2800682-18-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 2169220005 X-Stat-Signature: txsrz7btnipdgi17jjhs1qqogq5b9trc X-HE-Tag: 1672523182-387163 X-HE-Meta: U2FsdGVkX19NHhN+yIqPtrfmwH83LchZhWw0qJ+lPgIvWY8aykeXmlLLrI1mQue8RI1I16wDqSQTk1yw0RV152hn+6Ohgx9S3U1/yseP8k9uqD3kcYotsg04pHwoBSIlgaXSS6PImUQOBSEWc6nlyRWAI000zS2Qz9x/qTeIkpZsI1pd4+aWFE1LRBnfSYiamZRbGXkiT7IeF2LEbS1NtS/KYNkjWzgsoNeKSlW9RTdRjtZq/UfMLTIPhq2XjU/IjLGSXgaslcnLM8M2Yvqv7N/raXPuNz/wGXgihREowYdADy1KbbSThR5uY1gTuZLfiVSBAGIsL2YUQuYSDaGybJikVaVLpQaNUMzD53hZoVgv5917146zUBjngekw+Pk7guUmR4lguxvb/tKYT9CbNuGwQaNFvKXwmHQZafUD4sfxv8F5TDPwZ2+3+nyNNkZmMv3javao40pBmxq6UKZhPg0536vICrL9vuGVxnMtGgck/WT+SfKi5AeCpANwawHX/dHZB0z6R6k8QpiDOunaHgvMl8GpC2LZg6hFAW4B0liZ1tUM2Ee5de0GEXCHiElWKiEMFOgNjNvC0mJmSTzdoqZuUa+KAfX9i61PHoX9ESjuiQ9KiIgrzG3ss4RjwvUXbOe0BLHq99cYzTjkNyD/6H6Wje+OrochEzxBeHoa05JKes0Lz2cYNIecTHuPZOv6VK1AFbZH+7c6JPWp0/SvfgE4odN0Ux48mO073MKqg75utGzjxHlxtHEKNMcIR3TuPsAoSNO+t6UULrynxbx7LkGKKfDnPp04i0bEDU1Flv3nx44IATcnAjzIuLvTXMlyZHub2ElzBvtlajWeHag+Wv8egT7INAi6kbCympByxlM7ysXL51vCDS0yDwL7Go38Mgdu4Zep5EwOEU6bkg55OY6kee+m9l0Hwbvv5wYt1yQl8TmJ3T6a1g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Turn compound_nr() into a wrapper around folio_nr_pages(). Similarly to compound_order(), casting the struct page directly to struct folio preserves the existing behaviour, while calling page_folio() would change the behaviour. Move thp_nr_pages() down in the file so that compound_nr() can be after folio_nr_pages(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 38 ++++++++++++++++---------------------- 1 file changed, 16 insertions(+), 22 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 19bdb4b0e00a..75054fce8ec6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1016,18 +1016,6 @@ static inline void folio_set_compound_order(struct folio *folio, #endif } -/* Returns the number of pages in this potentially compound page. */ -static inline unsigned long compound_nr(struct page *page) -{ - if (!PageHead(page)) - return 1; -#ifdef CONFIG_64BIT - return page[1].compound_nr; -#else - return 1UL << compound_order(page); -#endif -} - /* Returns the number of bytes in this potentially compound page. */ static inline unsigned long page_size(struct page *page) { @@ -1050,16 +1038,6 @@ static inline unsigned int thp_order(struct page *page) return compound_order(page); } -/** - * thp_nr_pages - The number of regular pages in this huge page. - * @page: The head page of a huge page. - */ -static inline int thp_nr_pages(struct page *page) -{ - VM_BUG_ON_PGFLAGS(PageTail(page), page); - return compound_nr(page); -} - /** * thp_size - Size of a transparent huge page. * @page: Head page of a transparent huge page. @@ -1754,6 +1732,22 @@ static inline long folio_nr_pages(struct folio *folio) #endif } +/* Returns the number of pages in this potentially compound page. */ +static inline unsigned long compound_nr(struct page *page) +{ + return folio_nr_pages((struct folio *)page); +} + +/** + * thp_nr_pages - The number of regular pages in this huge page. + * @page: The head page of a huge page. + */ +static inline int thp_nr_pages(struct page *page) +{ + VM_BUG_ON_PGFLAGS(PageTail(page), page); + return compound_nr(page); +} + /** * folio_next - Move to the next physical folio. * @folio: The folio we're currently operating on. From patchwork Sat Dec 31 21:46:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89293C4332F for ; Sat, 31 Dec 2022 21:46:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 181418E0017; Sat, 31 Dec 2022 16:46:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DB718E0013; Sat, 31 Dec 2022 16:46:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3AF88E0013; Sat, 31 Dec 2022 16:46:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C131C8E0001 for ; Sat, 31 Dec 2022 16:46:36 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id AA8331A0572 for ; Sat, 31 Dec 2022 21:46:36 +0000 (UTC) X-FDA: 80303935992.23.A0F0762 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id 38A7E20006 for ; Sat, 31 Dec 2022 21:46:34 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=iFptPSjy; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523195; a=rsa-sha256; cv=none; b=Vz5CakRA6NyJFhF1+SC4PaKG6JmZW57BdhLsTJ+/Zym0bSfSNvbCngBGhehuOF3f3kSZBY dYyHahxvmf84RaC3qbWO7BvmOTulmoMg6WfRO38FCOSgQOR4Z75Q/lI+y2CeW4vrj5VWpE 91NYcGXiPsTksctuv9RbDVN/mzO4Ud8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=iFptPSjy; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523195; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Jm/W9DQTADB2OgGGthHKKMzuYVzV1Z3o9PGI+apsPUU=; b=HBSwC0pX35egPvDUhTQUdXjwQw5Gp/BZeNA02J3qUoFiWhwy+Q1+oH/qpY6cSDjZSGV+fb K1XfRRdKgNTaapRDLcNQSB1NEU8k3VQLlzFsaryYwhd22EiAjvQhJ+zg8l7syfO4uBPdJO lNht+x9g0tdk2EScuz0nbJPtlqxTQlA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Jm/W9DQTADB2OgGGthHKKMzuYVzV1Z3o9PGI+apsPUU=; b=iFptPSjyq1Xqn54VTcIyoUdqjj yF+GAuiw9ndR41RkC8eOf9+eLXQyQeuoffJkP24vwMgRh0mVxrsrfybJIbofTTCzFPczjCm9MKv2W Z+nuVNBEa3dUok9oiWhbKBdrf1xjvR0t4YVswsFs1JI/Y/EYrcmSQ+BB9vsOxr1dG3T5Bpcc91qN8 Cw/nKiRuSrhjaY/l63V26rj6vstrann4SI7a82qUQh0aKmIuXW3KTi/owER0xKZ9jtb+MQKis9e8W kgu6ZLFQCFRsfv/cJir4i6eeLZFPLfALPAPsErlCwkxTQj7WDkmzS1WMZpfja6SdRkrgqCUwpQWic ZMnDpf1w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgE-00Bkb1-5H; Sat, 31 Dec 2022 21:46:14 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 18/22] mm: Convert set_compound_page_dtor() and set_compound_order() to folios Date: Sat, 31 Dec 2022 21:46:06 +0000 Message-Id: <20221231214610.2800682-19-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 38A7E20006 X-Stat-Signature: bqpdi4y4cwtwxtuzwyj7pa5yftp6m73m X-HE-Tag: 1672523194-429106 X-HE-Meta: U2FsdGVkX19wlDSJv6aHjb7Tc/zfAgAqV1jfpLnOzQzd1SggGQhxUW/eIPXnpSo67WNEJ6glH4ddEBT0djy2q1kY4nqgH8Ok448ZMRKmyf0kah5t8HkL64QbilMAnvbSBvjnp7768W0GdDFxGaqq5iMGw2d6qeMo9GlW4ZUVr/pFNIXe/JoqoJbdaG0FreA0Cu4EOdXiH3YPz9/3Ijy65Z80xSBDerKdxYYnvrOF0r4RruyFFdMUON5Km+7/UWSvPpT0o9a5dQPNfsc/eOvjjomynZ3nxxz8Bf7F5IYMXwdUGp8UQ/V12gKf+QqPcnqYpFNCKZa1kjx73Wfyq7TFhTq1mt5kAnLIeRe9hCtQb+SJvAi9QLwSu/VRkXAsvtZ2K+i5zQYfMNli0e7RyKddkQeyR5wthDNr2nbuuGwXKZJ+I9CZtpdbWX529hOvgii3fEvUqBLddKdjh8GAnU58C92edRDZlbR+Eh1FiJzjqiGFuA1RlsCYmwGhA+mgQVv6bFbDSkPHloc8FGIrA54OIecVhDC2IEs2A0ipdATEvdoa8xnya2rfrttyPJsCRFaUJ5oYVpammkU5V7/cusmkSpx4f7d/wL4CAd3RNtqOhPI+otFPRA7zSzK/W8kXch7judlA3oCAmagLWEsu+jSh+v1K2I095rZdIxTe5KFzK5O1fmnm2vWk2xxlBOhH1aNoz2Zf98+kNf+8vITcQpBvvbRre2caCtMobsdFyMXtYk/ps+Qn989S4vFwFEQa6tFZ0cGHwTaPR4vM+rnKrzE3FinkKal+qt47V9ZQ38EDHZhwSgllnilxJss3H/v5H3dIMcoeyxIN0Tn9HBOnt2YSLrHPnbbTsicrGfsMIotQw4zqjKjUrmEUQJTVP6RW3adXt+I8lJwP+m4naD4djYAMA6eRNZo07Lhy0+2taesYaovSWq6iv3g5Xw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace uses of compound_dtor, compound_order and compound_nr by their folio equivalents. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 75054fce8ec6..125a9138d754 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -979,8 +979,11 @@ extern compound_page_dtor * const compound_page_dtors[NR_COMPOUND_DTORS]; static inline void set_compound_page_dtor(struct page *page, enum compound_dtor_id compound_dtor) { + struct folio *folio = (struct folio *)page; + VM_BUG_ON_PAGE(compound_dtor >= NR_COMPOUND_DTORS, page); - page[1].compound_dtor = compound_dtor; + VM_BUG_ON_PAGE(!PageHead(page), page); + folio->_folio_dtor = compound_dtor; } static inline void folio_set_compound_dtor(struct folio *folio, @@ -994,9 +997,11 @@ void destroy_large_folio(struct folio *folio); static inline void set_compound_order(struct page *page, unsigned int order) { - page[1].compound_order = order; + struct folio *folio = (struct folio *)page; + + folio->_folio_order = order; #ifdef CONFIG_64BIT - page[1].compound_nr = 1U << order; + folio->_folio_nr_pages = 1U << order; #endif } From patchwork Sat Dec 31 21:46:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4CF9C4332F for ; Sat, 31 Dec 2022 21:46:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 488F38E0011; Sat, 31 Dec 2022 16:46:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4393B8E0001; Sat, 31 Dec 2022 16:46:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3013E8E0011; Sat, 31 Dec 2022 16:46:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 179CB8E0001 for ; Sat, 31 Dec 2022 16:46:31 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E4309403ED for ; Sat, 31 Dec 2022 21:46:30 +0000 (UTC) X-FDA: 80303935740.28.E26D311 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id 7A3D51C0005 for ; Sat, 31 Dec 2022 21:46:29 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=A7IvrsOL; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523189; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yJp7zve7CcqmBnRWABrRT4eBqr9pxqmlqmYPi3FM1t4=; b=RxNzVXOwBfZeuDzN7HhWiOoDvsEN9EKbc/vgSBAfJLA0N6X3eTExWJ1VRmcJ+VNAljMfgh Xop4gV1DdFtqBJTqJX+oNd2aBJOjfQTZcqi+hHPsPsbuVDv66c8PQmjV2O8f0H3hxRpUnq HJrgw0OZMW2/jTJTGaAAZV2a79AwZrM= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=A7IvrsOL; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523189; a=rsa-sha256; cv=none; b=w7USCT50mKvSaLQRb4g4XamyBdaJLVltS0+CqOdW3cENEFk/GeNuh/uqV1WVnUgp4jJKmn WgSX/cgzpRrKSsx7w6CIRwTomjcst1mY8FNZr42l+8bRImRLTuy4nCrBJhqNUr7gCQu0l0 +htGt8BjQSDsAQknMoPpAJBQW+x23H8= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=yJp7zve7CcqmBnRWABrRT4eBqr9pxqmlqmYPi3FM1t4=; b=A7IvrsOLc+49SIhBXNSsHUV1r+ v+GRnxOhMCUFddh71ltSHL+Tkl2rYb2mZu0UzjKxLfHuSRqRlFJyS0xQB8lqiS4fMR2dUj/zSt2qp udfJpLFgAzs/aCdX0hRFmr3o8gyMgCdvx0BZrKYUZYeJR8r6eIbto4sEevjdN+cc1KiQ4rNBGsFcz urjM0sJMp2LWidCg+RmHtt0r4+v1yKykit2yAJmsLtUQ9qH4vi+OXsc+kDa7UxrCq1uTRXlQOxOwh 2c23k/mAs/f9DtL6WMmzBpZSxl1pRIysv3N/mVLiLVyorzRN4D/mx4sqI1afOFNgrWc1z8tU2UPJi LpwJbdWg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgE-00Bkb3-80; Sat, 31 Dec 2022 21:46:14 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 19/22] mm: Convert is_transparent_hugepage() to use a folio Date: Sat, 31 Dec 2022 21:46:07 +0000 Message-Id: <20221231214610.2800682-20-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7A3D51C0005 X-Stat-Signature: q5wj8kjjfy9ji1ukpc8q1tj5ocfntq9r X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1672523189-221090 X-HE-Meta: U2FsdGVkX18nvGb9bJqWSKxHxZ5mLMn2b69XB2HJu8LO2i5MGl5kw1KQz/FSnRaIaWbVYGnyj3HsCIiDfcFguXGQX3sFeuz4k70PG+BfxqnPTA8DzoM4JnHp6s1S2Xrk+yHLwyy2eWEi8/lsNyMiJEj8jpVlfuDHUMMR5YilgZzv1QR1uC8O/tsijmJrQJAnebHn4vOzztDVEwyKvNvqBXKbnPKhfz2DvhVjsgSiR6GVbeiefTIASanZXicK3PsvitE8s+V8Qhc5h5czMrd6p8BXaX3FY0oho6mwre1EeUGfXkAey1a1xJEDUKFPSotzoIjje3v9s73eWPxS59Optxxr8THkiYJhNvBbdRdlzTggFRtjmrsGYk2r3aHuI7g5gOgOAQJ2Kxd1HCJVy52MfraGsp8MtkiMcsHlPGWLenWoOlxpY6yMn7zu48FFO61vNMfVqnNPsUFufyvUTirLhDaPUaHaWYTRPQaRq3uHRN3DXOzeypvwN5nBnVmpT+eve/XFu9Y4Dg/Kiv/wDtiU0ehqACkzQQzDhGHVLH/ly0AERDOjl8fbjcUCk225TSo3kTkD8/KY9UCUniK/Xep53LAs88ZeyrkPlwrX2YZxPUXCIi5ZjY5Rk6dbppxRHd/rzRsUBQ70XLQimFr3O3MbggfWb5Wok6ZHao5jJlwUzxBwVOOSsUmAJZgBUSkWLh55VLi06eZyK7BfBDEaQpKrAAJA6jIKNypgCYGatSk5rq3KLr0VSZgSsXOwI4EPN0sEV4uivU0vxDM6Wkzakah6CAXwYSTZT3oJTlHwfQgGwzwphIsazvyvXZjjDZAKR4CQDoSHWNaZ2eMKUkfbSa3qsTEtsBGHK0rA4ZNG5eANlLIrpgdqW1QKQvGBUdxu4vqIfIl70OvMkLlSvek1VfjlfH0fCXKjHZEPEKP1R7TCoaooo7uc1HOrbg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace a use of page->compound_dtor with its folio equivalent. Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ca2eaec84726..c3a0964cf127 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -591,12 +591,14 @@ void prep_transhuge_page(struct page *page) static inline bool is_transparent_hugepage(struct page *page) { + struct folio *folio; + if (!PageCompound(page)) return false; - page = compound_head(page); - return is_huge_zero_page(page) || - page[1].compound_dtor == TRANSHUGE_PAGE_DTOR; + folio = page_folio(page); + return is_huge_zero_page(&folio->page) || + folio->_folio_dtor == TRANSHUGE_PAGE_DTOR; } static unsigned long __thp_get_unmapped_area(struct file *filp, From patchwork Sat Dec 31 21:46:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086175 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2E81C3DA7D for ; Sat, 31 Dec 2022 21:46:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28E058E000C; Sat, 31 Dec 2022 16:46:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F0208E0001; Sat, 31 Dec 2022 16:46:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC1858E000C; Sat, 31 Dec 2022 16:46:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CE3BD8E0001 for ; Sat, 31 Dec 2022 16:46:17 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9CE7F140169 for ; Sat, 31 Dec 2022 21:46:17 +0000 (UTC) X-FDA: 80303935194.15.4B85307 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id 3665240007 for ; Sat, 31 Dec 2022 21:46:15 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NRHJvStW; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523176; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VYgutuKC1jULeY5gjN0MKF6k95wzNsSotowhLphboKQ=; b=jBDEB0mJMFOOs5zcdpNqnRPxLHmNYUMVtV3jLwPEywTSDwh0Z4Kd8vqCnTXAoSOmLjYh5X BcFdewnFiZrkmbCuGin6l7VUvVbx7M1AxdpJaq4u0jRRLyHA+xDH0ilV90fSKwemKFgvLV FbdFz7RE+YGv7gv8fZIsY1+2NR/d6Ug= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=NRHJvStW; dmarc=none; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523176; a=rsa-sha256; cv=none; b=NVFhKrFBMOora4d7Zn1tdh8Klnorzp44NBcoszuq3TiXbLvkrBaiS0qyTRPTJNfKHvePhg pUuHPndaoXUIYuKCQ2JTaRh34/3eOZxEGGllePpKaZSCGpoG26C1zkj+el8wiLOv4s6ce3 HXQBUdgN57z9f2h4YUv21NSY7ORfB7o= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=VYgutuKC1jULeY5gjN0MKF6k95wzNsSotowhLphboKQ=; b=NRHJvStWhPJj8rlbd54+FRT2SX S3IOS3MukheSQOHEhiXq2Hi8Hwt23IpVgQYCJVrz9zhviwczt5o79N2AstN0v31ecrpzgG4LCeEUk 9MAx+/F3Zy/pAe84GSmI3cj5GDGEUT4j9Ru9VLw+eWYQqCG0RMoFrz2aNY01oYml1uQkAqu1FkaJI eQyCz0r6NCL9+74qBMeMQ5Mmv4JfOX5uVLJRf+rPWsvFhVw36i+VIUq27OCpY3OtVHWl9fQUeSCdT wDq3+A/HPZ6Cjwy7ae9ipUeoNYoEFYvB8c0LTXP0oRJW/ntYgDBoBjh66GCWeXQPBdpjbJf604L8N MfRsR6NQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgE-00Bkb5-Ag; Sat, 31 Dec 2022 21:46:14 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 20/22] mm: Convert destroy_large_folio() to use folio_dtor Date: Sat, 31 Dec 2022 21:46:08 +0000 Message-Id: <20221231214610.2800682-21-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3665240007 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: euenuchc1q5heijoio89yfkuopgkmg9m X-HE-Tag: 1672523175-150772 X-HE-Meta: U2FsdGVkX1+y3vgP51X+VIsxC/AFo50VyBTeXQJ7pWTxL9Bw5frs2C6tqJQ8ymg7wuEMha2b0HyFRldoIcfoPfXtyQCxgb1x8p8mmIpBFTxkzM7BdEB4zVkLWElzos5Aed0kMhvazKjbNMjK8Q2c5dD7LaGkBNdvQRP9/oKrYcxF9ARGWwbI0MbAWiHoyAvmdFeIHBC7SJz11G4KjXk7eZhcdoxRhQEZ8tiUtalOzdZO+u2hWaIIw0L7ii04nVCyg+abZQpi7VoUrCvDuWv5KiNrxX4bhgHT68nkML82U8jdmKSIq3ByGginSoErNoY2bF7HpnNGa5SIQCd60ht7zbA9mXr+W27A0m3mJxDJCBwWh47nfAncyZefeUXOc1ZViGYYS98+NPVzoIWx+1JXeUI+Ow8xRTLx40gN3+Jt/TbAbA3T1G/utL+bibeLAQJ3FX4VNhEjh2bQD6KV1ENxVW7dR+AWcvDVZV9mzj7flVmIi9r8ckrP/nOQq7fC0jM6FTXyQFgDUYWWVW6ny/FV0MWkyc/YZ+nHBI6iqYw4/VWEzWeZxWHLM1116325ye0kmps6wYCfBIPtsjTSBHCaFcXBr668avLV+WthT1BNWuJcVc5fmYwjHrCxgKuUKq/oV9r9cdN5ASfCXyfBEvuWDUarHNjN/f//vtcAWXQUno6pHd6sSeka43tXDM8ORl+TnE09vm9zciL0lou9+/jcuHTn/TDHns4rdflxmzt0nGZoR4R2+VYqPzPMzh2ct76zazzthoozSN8vvhekZ73M/ZXhpO2ZcaGBl7f5DOAdEjPlRcGsl33Cbhk1LXWaz/szJnGggJPtad5FgT3OxbM2A1nBa+kWM39dPp/vcdWpeO4sj+ydWuklktOCq+5GMequFjtgekzY05KDnxJR5fZD/cfxoUnyL0lFQDp8SAixPOqsMTWXtXH5yA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace a use of compound_dtor. Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d309c48afc52..5f24eff14b96 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -807,7 +807,7 @@ void prep_compound_page(struct page *page, unsigned int order) void destroy_large_folio(struct folio *folio) { - enum compound_dtor_id dtor = folio_page(folio, 1)->compound_dtor; + enum compound_dtor_id dtor = folio->_folio_dtor; VM_BUG_ON_FOLIO(dtor >= NR_COMPOUND_DTORS, folio); compound_page_dtors[dtor](&folio->page); From patchwork Sat Dec 31 21:46:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02483C4167B for ; Sat, 31 Dec 2022 21:46:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F84F8E0013; Sat, 31 Dec 2022 16:46:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 830848E0016; Sat, 31 Dec 2022 16:46:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 354CE8E0016; Sat, 31 Dec 2022 16:46:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E464C8E0015 for ; Sat, 31 Dec 2022 16:46:36 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id BBA2D120308 for ; Sat, 31 Dec 2022 21:46:36 +0000 (UTC) X-FDA: 80303935992.21.627E2A0 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 5491D100008 for ; Sat, 31 Dec 2022 21:46:35 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=n4kXmngh; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523195; a=rsa-sha256; cv=none; b=HXFT8qx2L4zwVg23EyqKBKOJf1BX/fesTuIdV/ZF5WiOPmuLz/qAcZ+33QwXLU5e+ITe7b 9iKTkiMG/mUFczlj693CO5qqU7WLH01Ig+cfJqZE94YeCv6mqMHh2tW2G1fJikyONlw5cm +p17eAHuQxYBAWBwSXalciFCOZiwPVo= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=n4kXmngh; spf=none (imf14.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523195; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CGchFUBsAib0PL0s6WPLK/pl946BBf2kZqDftY5vKhs=; b=L9OJjAPOLeLsbbULrRHdlthFB2LTjcuPeFYSMBnM2XRzMeRCCcZynuWCGPT2tbh/GW+VEt 1mdaZE4WOlr55r2vnGba/vrzb3C/aOy1NwY5BfrhyaSdXnX3CX1ijp0c1xFBJAyoxWb3/u wetKsW2klFG19yD/kLOL/cylIvfo7VA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=CGchFUBsAib0PL0s6WPLK/pl946BBf2kZqDftY5vKhs=; b=n4kXmnghsKrF5WGPpiRakbUr3K 4fxjDZCauHlNFELKsNJ2SQJ+RL4QFCtXgUIyTUP7PchLyQtao7O4lGZMUYWq5PVJ6R2bfmPRU+7nI jM8lQXnRtdNW5fwghNSpJAtcBFpb4tTfSVdG1Yq+6aEca3TS2G+Z7bbeWq94Yh4FwjG+0n/E1eutK TfsTHjZcpY7dnaUW0cqZdWq4kpYiN6LVpn4iXAFgIR1rYDzi4ujNQ+qiLoX36oUwBYwNiNl9XelOa 78gHMx4kEOJYMCGP2v7NGUTx8a1ZiQu41ZZ+WGGwLCc86UqKYH+89BdeJC5EBHoSseRvHuD2yNrHE hn05O4ZA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgE-00Bkb7-DO; Sat, 31 Dec 2022 21:46:14 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 21/22] hugetlb: Remove uses of compound_dtor and compound_nr Date: Sat, 31 Dec 2022 21:46:09 +0000 Message-Id: <20221231214610.2800682-22-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 5491D100008 X-Rspamd-Server: rspam01 X-Stat-Signature: 7wt6b53e8iifigx1r59oz7tfzfnteqns X-HE-Tag: 1672523195-661917 X-HE-Meta: U2FsdGVkX18AjmHsYOc1fI2E7NO9YG/em6Oius2YWCDHQUiy2lKekXSp3+VLyF2ZG24+seZL+SMXsvHMEd8lz7aDKcAEe2/kiqm0tU24MHLLkD7+fSvIlU698GjS18bJLuAjUTK7z/Ebmk9ImM+rZjT7SUxvkL7tGsd8j1fSlz/btVS4Ir9zXIPVIqC9XNp1pGk7DIsoRrexrXJp40cPES/rYA+sH3VKOV6O2J/mxx+xpVw1hAmuoV5ijp6G/gET06cAX1gg0KvZ7ongCIQjNfHgFsznOx3BZUbVMK8ju7cNTebkqG2pwzEjgLODF9TL9oeDlmZlZQQ0lLyb49Iv2kDwstqR8cihGqm/x7QZA9U9yc7Nakb09Hx3E5Ek95W+X9GlxkqaryCiucwIPQmZEVazWhHU+rr9sOJvkRXbZ6O2Mers5DeYSzi6lQ0pEaKAAXS4INinxd4a515+odfj9yV4r1Eua5ITwE5BaZxO3Mqmo5Mjh80wPIs/eZJH5g0TnTThYMDiAlkNhYejLHMw+HFfy6wIpfgSgpthFGDnnF42wg3I4H8eGPnT9+EXqMF7U7ajEtB9+I+PuFLY3rULaBt2tljshPCAhUjZYiMZlM2gy/Gqms3zj4bZbX/3FBDBsOn72xI9NF3zPmOFCRu/xsAfsGr7+b5Gx0tcIkrLsdxlxYXDVDyBxypluruwUTDX4nhFnV4d/8Venb1hmfG6l3qY67qyYQpN2bLnksy9Bo5Zpd6VSLFUmwkBPeS/hJC1Bvr2NfJKF1r/r+an75XmmWEkAe3DJ6LhuxzH8yWKDBz7tbXNpcSKNm+6+TPdnIrNdVcKCMH/NM27oAZ7gtBZN1W70w969lL6G+zOzp7N2Gv+6+VWFbRca2WON7IRNsk+MVCKKuTPDhBAQd6y6JY2dBBr36mqk+jdIpwXkAzhN0dg8kFQMkmqRA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert the entire file to use the folio equivalents. Signed-off-by: Matthew Wilcox (Oracle) --- mm/hugetlb.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0a6bb471c1a7..72f21bdc581e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2042,11 +2042,12 @@ static bool prep_compound_gigantic_folio_for_demote(struct folio *folio, */ int PageHuge(struct page *page) { + struct folio *folio; + if (!PageCompound(page)) return 0; - - page = compound_head(page); - return page[1].compound_dtor == HUGETLB_PAGE_DTOR; + folio = page_folio(page); + return folio->_folio_dtor == HUGETLB_PAGE_DTOR; } EXPORT_SYMBOL_GPL(PageHuge); @@ -2056,10 +2057,11 @@ EXPORT_SYMBOL_GPL(PageHuge); */ int PageHeadHuge(struct page *page_head) { - if (!PageHead(page_head)) + struct folio *folio = (struct folio *)page_head; + if (!folio_test_large(folio)) return 0; - return page_head[1].compound_dtor == HUGETLB_PAGE_DTOR; + return folio->_folio_dtor == HUGETLB_PAGE_DTOR; } EXPORT_SYMBOL_GPL(PageHeadHuge); From patchwork Sat Dec 31 21:46:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13086177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15409C4332F for ; Sat, 31 Dec 2022 21:46:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 574058E000E; Sat, 31 Dec 2022 16:46:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 500F88E0001; Sat, 31 Dec 2022 16:46:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 302A68E000E; Sat, 31 Dec 2022 16:46:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 089598E0001 for ; Sat, 31 Dec 2022 16:46:22 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D86F28023D for ; Sat, 31 Dec 2022 21:46:21 +0000 (UTC) X-FDA: 80303935362.26.9644B3A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id 581AD1A000A for ; Sat, 31 Dec 2022 21:46:20 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DBuTMUN1; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672523180; a=rsa-sha256; cv=none; b=237w404C63V7lpD5oVkRoqMqR+BNP3q/3EQgTNcsheVvH+Ne4pQJLWeAepeW3eDLtg3GHl xfiRPz8urkcG7+QCjuNQmdXgjjvoxl0ccpCZtLUBfIjNaSfb9e/mI47Q9pdFjtgdwZCSdZ IsHuxavu6EL+Ng+2t/Wr+yHe3s7uAwA= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DBuTMUN1; spf=none (imf19.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672523180; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JaUOuTob4FRDJ2LnlAQkJD9Y/TDngA5NdRLNJbcA8pE=; b=cbC8Sy1qkPWzFbrED/m1mbyXzymCjucV2VSFcno982QrcYoYdzqAwz1ZPW1f6AMqpigC9i PNCZOo+n7Y/GerqMNSc11H0LNm2K6ujg7iXirNZoqPW49lFSfhV6IT9kR0oDVyTRzTJv5Z otMS9zwjVQD0Xk3H8Qm7u7wmIb3eFjQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=JaUOuTob4FRDJ2LnlAQkJD9Y/TDngA5NdRLNJbcA8pE=; b=DBuTMUN1rWw85o2t2k4ATGQRhw PTv8ZjZcHnwveBA5P+sLZlFY42XkZw/Uo01CTTOqoRUKOervwtqbWD//ct2ulFMW/CaMD1IV4rY9H D6IJ+kboeY1Ar/Lg4x4l8uQmtV443MnKnyFRtQM9lGS0ggquaNQJzVAdnuhySIYNZDmC/WKs02Vq0 eOdRtdwVrLkA3blXya6HYQpZTf5n1Wg65zzSOFt8Lzcewckz5Am7qnC58MyVblJI9K6k9gbqNtRDe spI44F5ILQff66znHmlADBobgyKuIBlNefKDQLm7BnBhTNalILyefQHIUtYuK50dbR8mo3RV0Eu2i ulR0dsMw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pBjgE-00Bkb9-Fa; Sat, 31 Dec 2022 21:46:14 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, Hugh Dickins Subject: [PATCH 22/22] mm: Remove 'First tail page' members from struct page Date: Sat, 31 Dec 2022 21:46:10 +0000 Message-Id: <20221231214610.2800682-23-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20221231214610.2800682-1-willy@infradead.org> References: <20221231214610.2800682-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 581AD1A000A X-Stat-Signature: qrw4rckt7w8qzogdki8stngmbifrjgpd X-HE-Tag: 1672523180-37397 X-HE-Meta: U2FsdGVkX1//Bd8XQRoaFVGBanTCFQjPm25VIMhVryaKTpicdFgLUAZ4jT1EaJ1Z66+UY0jtCoq06JunzvqP9PAZzRg5JZzfqKvNElVgGDHQNcxJHwue4w6VuAuxKU3QZqOLQUEhXszqBTFEOc9gTJ8sVulFVMqh3XgGHlvaDRJ8YGcIYnNzBaS/82/Vi0Dkn1gABACfNbUalDr/SxJ7dh/4CdwfDfqcT/vH0B8/cvWtERs3jdsoANY6jSkgCegwzW+9cbXteuOIrRuMQsd6B8HPXMDgyMRtV4i1uYO86RCfma2Wk0An0M5nzqSUmtITJgRAdicKHW+UqmKWBSqZxkZNDLVDuhBgw69Zhu6GWsRIKB2b4yCbdYAgSHM13w7eoOsao+TvpSVZ8GBd5aeWKS9NrdGEXgyIlL5h/5JMxkVODXShN16Dlr/quZ7EtL9a98dvnWoaotnZNXaHE0M4FAtvwESoja0RVVMFeqnCKGuDOFMlgZQHteFeRlMrjdQk4Y0Mw0bxYqxuzTS0MlnqyZGP+xG5+Ody+H2q3d6iZK3bPidZEWBC83Afkmvv8Njmyv+a33HE4OGSbSNAH8rSMNY4ejA6PlDBfOJmpNDY9QxACLhhcucCbMASLksS6WkGtnb+W+ykq8fiGjFipDPCJSZvKRvkE3hwnVclG6H47DJlVLJGHeedkkuZ/Mlm0OD3+99OtohBe7Dki/7bhuCUHLAe+r2FGh9uchGVrgeuAYTfwift0X2ivGcTXOXgWBhwJPjYZhztOv+AADv8Na6Evx/4uhFTICtl0BI01JvArNSiEVxEpo4JG8Kxk7fcLZdReKmuMYDi4PKzLHhmavI3XwOh6nn2c1RuPG8iD6xmn9QH9mjrW2EdTRTsr4fZv6TCT6r51r2B410= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All former users now use the folio equivalents, so remove them from the definition of struct page. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm_types.h | 18 ------------------ kernel/crash_core.c | 4 ++-- 2 files changed, 2 insertions(+), 20 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 381103c81cdf..de879a9f74a8 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -140,16 +140,6 @@ struct page { }; struct { /* Tail pages of compound page */ unsigned long compound_head; /* Bit zero is set */ - - /* First tail page only */ - unsigned char compound_dtor; - unsigned char compound_order; - atomic_t compound_mapcount; - atomic_t subpages_mapcount; - atomic_t compound_pincount; -#ifdef CONFIG_64BIT - unsigned int compound_nr; /* 1 << compound_order */ -#endif }; struct { /* Second tail page of transparent huge page */ unsigned long _compound_pad_1; /* compound_head */ @@ -401,14 +391,6 @@ FOLIO_MATCH(memcg_data, memcg_data); offsetof(struct page, pg) + sizeof(struct page)) FOLIO_MATCH(flags, _flags_1); FOLIO_MATCH(compound_head, _head_1); -FOLIO_MATCH(compound_dtor, _folio_dtor); -FOLIO_MATCH(compound_order, _folio_order); -FOLIO_MATCH(compound_mapcount, _entire_mapcount); -FOLIO_MATCH(subpages_mapcount, _nr_pages_mapped); -FOLIO_MATCH(compound_pincount, _pincount); -#ifdef CONFIG_64BIT -FOLIO_MATCH(compound_nr, _folio_nr_pages); -#endif #undef FOLIO_MATCH #define FOLIO_MATCH(pg, fl) \ static_assert(offsetof(struct folio, fl) == \ diff --git a/kernel/crash_core.c b/kernel/crash_core.c index 87ef6096823f..755f5f08ab38 100644 --- a/kernel/crash_core.c +++ b/kernel/crash_core.c @@ -455,8 +455,8 @@ static int __init crash_save_vmcoreinfo_init(void) VMCOREINFO_OFFSET(page, lru); VMCOREINFO_OFFSET(page, _mapcount); VMCOREINFO_OFFSET(page, private); - VMCOREINFO_OFFSET(page, compound_dtor); - VMCOREINFO_OFFSET(page, compound_order); + VMCOREINFO_OFFSET(folio, _folio_dtor); + VMCOREINFO_OFFSET(folio, _folio_order); VMCOREINFO_OFFSET(page, compound_head); VMCOREINFO_OFFSET(pglist_data, node_zones); VMCOREINFO_OFFSET(pglist_data, nr_zones);