From patchwork Fri Mar 5 04:18:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43172C4332B for ; Fri, 5 Mar 2021 04:19:19 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C312E65012 for ; Fri, 5 Mar 2021 04:19:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C312E65012 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CDFC76B0008; Thu, 4 Mar 2021 23:19:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CC9D86B000C; Thu, 4 Mar 2021 23:19:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF6BC6B000A; Thu, 4 Mar 2021 23:19:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0056.hostedemail.com [216.40.44.56]) by kanga.kvack.org (Postfix) with ESMTP id 930486B0007 for ; Thu, 4 Mar 2021 23:19:17 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 5ACA34DDB for ; Fri, 5 Mar 2021 04:19:17 +0000 (UTC) X-FDA: 77884515954.16.44B32B6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id 8706FA0009CE for ; Fri, 5 Mar 2021 04:19:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=A+aKTxH1jcd4vDiytddTmJl7H4EV0ZPBIjE0cYaDNCI=; b=A4wG4jzqn9eHv9/2JTAvx+xA/U 25l66qpt/vLi9bDvZsHz9Zc3oJklrTMd2Qx8/HTrr6MrO4d9oq69lXC2c6GlXZ5akGhJBeKCvWudU 9Qncc7ku0kF+vAVL0yvGEFAfaiIzukYjYQ59Z49VvZIL3qRXCbtcHaMBYXXDrgOrjqLkLTRsmMBnw gs8as3GeHsTDPDENrkLFJQywPSh4JC1y9dULBcEowIIHUJbZMzGoJBi+2cyhFnERZLK1wFFhU8DjC AZgUqKCQQQSfvuURYACDnr3WUwcyvYDmmhbJqAwB4qriB3jednMywuMUxrNd2M6a/vlRfj6GDl/mM +XTB+qdw==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1vg-00A3SJ-7k; Fri, 05 Mar 2021 04:19:08 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 01/25] mm: Introduce struct folio Date: Fri, 5 Mar 2021 04:18:37 +0000 Message-Id: <20210305041901.2396498-2-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 7dxaqurxzaf8h51ned48oj7bx49ajugt X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8706FA0009CE Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf15; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614917954-706154 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A struct folio refers to an entire (possibly compound) page. A function which takes a struct folio argument declares that it will operate on the entire compound page, not just PAGE_SIZE bytes. In return, the caller guarantees that the pointer it is passing does not point to a tail page. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 30 ++++++++++++++++++++++++++++++ include/linux/mm_types.h | 17 +++++++++++++++++ 2 files changed, 47 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 77e64e3eac80..a46e5a4385b0 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -927,6 +927,11 @@ static inline unsigned int compound_order(struct page *page) return page[1].compound_order; } +static inline unsigned int folio_order(struct folio *folio) +{ + return compound_order(&folio->page); +} + static inline bool hpage_pincount_available(struct page *page) { /* @@ -1518,6 +1523,30 @@ static inline void set_page_links(struct page *page, enum zone_type zone, #endif } +static inline unsigned long folio_nr_pages(struct folio *folio) +{ + return compound_nr(&folio->page); +} + +static inline struct folio *next_folio(struct folio *folio) +{ +#if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP) + return (struct folio *)nth_page(&folio->page, folio_nr_pages(folio)); +#else + return folio + folio_nr_pages(folio); +#endif +} + +static inline unsigned int folio_shift(struct folio *folio) +{ + return PAGE_SHIFT + folio_order(folio); +} + +static inline size_t folio_size(struct folio *folio) +{ + return PAGE_SIZE << folio_order(folio); +} + /* * Some inline functions in vmstat.h depend on page_zone() */ @@ -1623,6 +1652,7 @@ extern void pagefault_out_of_memory(void); #define offset_in_page(p) ((unsigned long)(p) & ~PAGE_MASK) #define offset_in_thp(page, p) ((unsigned long)(p) & (thp_size(page) - 1)) +#define offset_in_folio(folio, p) ((unsigned long)(p) & (folio_size(folio) - 1)) /* * Flags passed to show_mem() and show_free_areas() to suppress output in diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0974ad501a47..a311cb48526f 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -223,6 +223,23 @@ struct page { #endif } _struct_page_alignment; +/* + * A struct folio is either a base (order-0) page or the head page of + * a compound page. + */ +struct folio { + struct page page; +}; + +static inline struct folio *page_folio(struct page *page) +{ + unsigned long head = READ_ONCE(page->compound_head); + + if (unlikely(head & 1)) + return (struct folio *)(head - 1); + return (struct folio *)page; +} + static inline atomic_t *compound_mapcount_ptr(struct page *page) { return &page[1].compound_mapcount; From patchwork Fri Mar 5 04:18:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 367B0C433E6 for ; Fri, 5 Mar 2021 04:19:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A1AC664FF0 for ; Fri, 5 Mar 2021 04:19:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A1AC664FF0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 180326B0005; Thu, 4 Mar 2021 23:19:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 10BF66B0007; Thu, 4 Mar 2021 23:19:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EECE66B0008; Thu, 4 Mar 2021 23:19:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0045.hostedemail.com [216.40.44.45]) by kanga.kvack.org (Postfix) with ESMTP id CFA8D6B0005 for ; Thu, 4 Mar 2021 23:19:16 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 8EE572461 for ; Fri, 5 Mar 2021 04:19:16 +0000 (UTC) X-FDA: 77884515912.03.34C150D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id CBC6A80192DD for ; Fri, 5 Mar 2021 04:19:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=/ziAJ/GI8dm5TexYAQD4UuVoJKdeAx1m/+HmyZTNY7I=; b=WggUtDfW+fP/+lke5TpqBPeH3U PvKUe5/NU7Xs+Sy5ioYl7QfiGg8dXPtgB5jjDiJGS9P5+Zpfg+akJtjl8zQwgflHtD/IiA3bUgz0F C03c26Dg0kTEXzPt8Yw+bbKaq0Tvrg0eX9NVyhOAG0EpzaBEA/fHzdNtvSCdAkfpCnTck7us8BDTT g+Sw9r1+oCrh2uMgK/nIXp2EZGXL4s329Bw/E47kMyd/e89zLE5U4a9hpd2vQpkwjMhaCqe/jm0iJ q0IkAWW+/VXoWL0Dy/ZZUtI1O+VfSey7SvdsfM1x1gLQLDgeeFBEdsfdFf+186G+CJHsfzSpw9SoC 59uyyamA==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1vi-00A3SZ-0r; Fri, 05 Mar 2021 04:19:10 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Zi Yan Subject: [PATCH v4 02/25] mm: Add folio_pgdat and folio_zone Date: Fri, 5 Mar 2021 04:18:38 +0000 Message-Id: <20210305041901.2396498-3-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: CBC6A80192DD X-Stat-Signature: z1w6akdxyxjuziqckxzyw131ux8y54kw Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614917954-345218 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These are just convenience wrappers for callers with folios; pgdat and zone can be reached from tail pages as well as head pages. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan --- include/linux/mm.h | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index a46e5a4385b0..ecfe202aa4ec 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1488,6 +1488,16 @@ static inline pg_data_t *page_pgdat(const struct page *page) return NODE_DATA(page_to_nid(page)); } +static inline struct zone *folio_zone(const struct folio *folio) +{ + return page_zone(&folio->page); +} + +static inline pg_data_t *folio_pgdat(const struct folio *folio) +{ + return page_pgdat(&folio->page); +} + #ifdef SECTION_IN_PAGE_FLAGS static inline void set_page_section(struct page *page, unsigned long section) { From patchwork Fri Mar 5 04:18:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BD79C433E0 for ; Fri, 5 Mar 2021 04:19:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1229564FE4 for ; Fri, 5 Mar 2021 04:19:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1229564FE4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A82CD6B000A; Thu, 4 Mar 2021 23:19:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A37256B000C; Thu, 4 Mar 2021 23:19:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 837136B000D; Thu, 4 Mar 2021 23:19:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0246.hostedemail.com [216.40.44.246]) by kanga.kvack.org (Postfix) with ESMTP id 5E11F6B000A for ; Thu, 4 Mar 2021 23:19:19 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1DA748249980 for ; Fri, 5 Mar 2021 04:19:19 +0000 (UTC) X-FDA: 77884516038.22.B75796F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 1CD75C0001EA for ; Fri, 5 Mar 2021 04:19:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=TfEJs96Ck13Yfn0mZfn2F+YPOt+7EPnxf8ur6JTtW6I=; b=cRV61d0eP1fedPSGYC6ZnAUwFS XtfS+nf5eiaZmnEnJSVbX/ArI+5Jxk4D+PP4MaR41DXmEWpdpey/mm30HlcSeETtWWuYeCA8MLaPg vrhI2uhim3+Xmu/JZGTttjrD3I2rzxICJciJ3vB4y25C2Z+aDaRaLkMoxEzThBoiVyY954V/Yzj+V BXK5Qtmh7STyqh3UcPJZ5KB7amOvBhsWy/UFdiszD8jv787MW7fjQAn7uYvaCopZwEQK2GMQD/uEc 7+qkSU7PI4XATdFC1B0rxyMxmhHveknKkQe7TB9YKMeNvgykMpnX8pLipQMskxj/eJ678h2IB4QLx MvfEas1Q==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1vk-00A3Sg-4R; Fri, 05 Mar 2021 04:19:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 03/25] mm/vmstat: Add functions to account folio statistics Date: Fri, 5 Mar 2021 04:18:39 +0000 Message-Id: <20210305041901.2396498-4-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1CD75C0001EA X-Stat-Signature: 6zukqejfuieenhwebjn3wc4ut8tmjt4y Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614917957-858105 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow page counters to be more readily modified by callers which have a folio. Name these wrappers with 'stat' instead of 'state' as requested by Linus here: https://lore.kernel.org/linux-mm/CAHk-=wj847SudR-kt+46fT3+xFFgiwpgThvm7DJWGdi4cVrbnQ@mail.gmail.com/ Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/vmstat.h | 83 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 506d625163a1..dff5342df07b 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -402,6 +402,54 @@ static inline void drain_zonestat(struct zone *zone, struct per_cpu_pageset *pset) { } #endif /* CONFIG_SMP */ +static inline +void __inc_zone_folio_stat(struct folio *folio, enum zone_stat_item item) +{ + __mod_zone_page_state(folio_zone(folio), item, folio_nr_pages(folio)); +} + +static inline +void __dec_zone_folio_stat(struct folio *folio, enum zone_stat_item item) +{ + __mod_zone_page_state(folio_zone(folio), item, -folio_nr_pages(folio)); +} + +static inline +void inc_zone_folio_stat(struct folio *folio, enum zone_stat_item item) +{ + mod_zone_page_state(folio_zone(folio), item, folio_nr_pages(folio)); +} + +static inline +void dec_zone_folio_stat(struct folio *folio, enum zone_stat_item item) +{ + mod_zone_page_state(folio_zone(folio), item, -folio_nr_pages(folio)); +} + +static inline +void __inc_node_folio_stat(struct folio *folio, enum node_stat_item item) +{ + __mod_node_page_state(folio_pgdat(folio), item, folio_nr_pages(folio)); +} + +static inline +void __dec_node_folio_stat(struct folio *folio, enum node_stat_item item) +{ + __mod_node_page_state(folio_pgdat(folio), item, -folio_nr_pages(folio)); +} + +static inline +void inc_node_folio_stat(struct folio *folio, enum node_stat_item item) +{ + mod_node_page_state(folio_pgdat(folio), item, folio_nr_pages(folio)); +} + +static inline +void dec_node_folio_stat(struct folio *folio, enum node_stat_item item) +{ + mod_node_page_state(folio_pgdat(folio), item, -folio_nr_pages(folio)); +} + static inline void __mod_zone_freepage_state(struct zone *zone, int nr_pages, int migratetype) { @@ -536,6 +584,24 @@ static inline void __dec_lruvec_page_state(struct page *page, __mod_lruvec_page_state(page, idx, -1); } +static inline void __mod_lruvec_folio_stat(struct folio *folio, + enum node_stat_item idx, int val) +{ + __mod_lruvec_page_state(&folio->page, idx, val); +} + +static inline void __inc_lruvec_folio_stat(struct folio *folio, + enum node_stat_item idx) +{ + __mod_lruvec_folio_stat(folio, idx, folio_nr_pages(folio)); +} + +static inline void __dec_lruvec_folio_stat(struct folio *folio, + enum node_stat_item idx) +{ + __mod_lruvec_folio_stat(folio, idx, -folio_nr_pages(folio)); +} + static inline void inc_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx) { @@ -560,4 +626,21 @@ static inline void dec_lruvec_page_state(struct page *page, mod_lruvec_page_state(page, idx, -1); } +static inline void mod_lruvec_folio_stat(struct folio *folio, + enum node_stat_item idx, int val) +{ + mod_lruvec_page_state(&folio->page, idx, val); +} + +static inline void inc_lruvec_folio_stat(struct folio *folio, + enum node_stat_item idx) +{ + mod_lruvec_folio_stat(folio, idx, folio_nr_pages(folio)); +} + +static inline void dec_lruvec_folio_stat(struct folio *folio, + enum node_stat_item idx) +{ + mod_lruvec_folio_stat(folio, idx, -folio_nr_pages(folio)); +} #endif /* _LINUX_VMSTAT_H */ From patchwork Fri Mar 5 04:18:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56466C433E6 for ; Fri, 5 Mar 2021 04:19:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E83E465012 for ; Fri, 5 Mar 2021 04:19:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E83E465012 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4400B6B000C; Thu, 4 Mar 2021 23:19:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F0446B000D; Thu, 4 Mar 2021 23:19:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B7BF6B000E; Thu, 4 Mar 2021 23:19:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0235.hostedemail.com [216.40.44.235]) by kanga.kvack.org (Postfix) with ESMTP id 084E76B000C for ; Thu, 4 Mar 2021 23:19:26 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B37BD1802EAD4 for ; Fri, 5 Mar 2021 04:19:25 +0000 (UTC) X-FDA: 77884516290.14.DC74E2F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id 2AF36A0009C5 for ; Fri, 5 Mar 2021 04:19:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=7x6Ty5MYTpzZNiotwF3DtVw5Sni26V7oFSAW6dNR/BU=; b=Wb1jSPKPO3K3fViSoGnK2DgV5d Ax0oAKOCehujoZ4ZEATVJC28w0SEsEwINRYnpBUoXisBj3XrGOwSa8B8E+AwRCenM+i6KMNVJArif KOqWKiiDrk143vmc9gb1d2KaNTH2neA9x2uIjUN8bqte7g8NzK2ejMa0f/4QUJuTAwPWe21BzjIWS GUCtLphocGVnHQ3ZvP1QG2V1IbvpqYWVbo9ixux5y9i1MqlGwU+C8FPOTvCmTz5DKjp0MeFw89ufn p5ilyvYhRJ3IyGz+hZWdVzf0p7stsd+XLWPGZCzyIVAU6ZGOmwub0pa8SF9/2ofgQNWh2JQYajK71 Ll28toNg==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1vm-00A3Sl-Co; Fri, 05 Mar 2021 04:19:15 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Zi Yan Subject: [PATCH v4 04/25] mm/debug: Add VM_BUG_ON_FOLIO and VM_WARN_ON_ONCE_FOLIO Date: Fri, 5 Mar 2021 04:18:40 +0000 Message-Id: <20210305041901.2396498-5-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: gikmgz4duczucehzitbq8anx3pnuykun X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 2AF36A0009C5 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf15; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614917965-964755 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These are the folio equivalents of VM_BUG_ON_PAGE and VM_WARN_ON_ONCE_PAGE. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan --- include/linux/mmdebug.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/include/linux/mmdebug.h b/include/linux/mmdebug.h index 5d0767cb424a..77d24e1dcaec 100644 --- a/include/linux/mmdebug.h +++ b/include/linux/mmdebug.h @@ -23,6 +23,13 @@ void dump_mm(const struct mm_struct *mm); BUG(); \ } \ } while (0) +#define VM_BUG_ON_FOLIO(cond, folio) \ + do { \ + if (unlikely(cond)) { \ + dump_page(&folio->page, "VM_BUG_ON_FOLIO(" __stringify(cond)")");\ + BUG(); \ + } \ + } while (0) #define VM_BUG_ON_VMA(cond, vma) \ do { \ if (unlikely(cond)) { \ @@ -48,6 +55,17 @@ void dump_mm(const struct mm_struct *mm); } \ unlikely(__ret_warn_once); \ }) +#define VM_WARN_ON_ONCE_FOLIO(cond, folio) ({ \ + static bool __section(".data.once") __warned; \ + int __ret_warn_once = !!(cond); \ + \ + if (unlikely(__ret_warn_once && !__warned)) { \ + dump_page(&folio->page, "VM_WARN_ON_ONCE_FOLIO(" __stringify(cond)")");\ + __warned = true; \ + WARN_ON(1); \ + } \ + unlikely(__ret_warn_once); \ +}) #define VM_WARN_ON(cond) (void)WARN_ON(cond) #define VM_WARN_ON_ONCE(cond) (void)WARN_ON_ONCE(cond) @@ -56,11 +74,13 @@ void dump_mm(const struct mm_struct *mm); #else #define VM_BUG_ON(cond) BUILD_BUG_ON_INVALID(cond) #define VM_BUG_ON_PAGE(cond, page) VM_BUG_ON(cond) +#define VM_BUG_ON_FOLIO(cond, folio) VM_BUG_ON(cond) #define VM_BUG_ON_VMA(cond, vma) VM_BUG_ON(cond) #define VM_BUG_ON_MM(cond, mm) VM_BUG_ON(cond) #define VM_WARN_ON(cond) BUILD_BUG_ON_INVALID(cond) #define VM_WARN_ON_ONCE(cond) BUILD_BUG_ON_INVALID(cond) #define VM_WARN_ON_ONCE_PAGE(cond, page) BUILD_BUG_ON_INVALID(cond) +#define VM_WARN_ON_ONCE_FOLIO(cond, folio) BUILD_BUG_ON_INVALID(cond) #define VM_WARN_ONCE(cond, format...) BUILD_BUG_ON_INVALID(cond) #define VM_WARN(cond, format...) BUILD_BUG_ON_INVALID(cond) #endif From patchwork Fri Mar 5 04:18:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85B3BC433E0 for ; Fri, 5 Mar 2021 04:19:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2C7D464FF0 for ; Fri, 5 Mar 2021 04:19:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2C7D464FF0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C2FBE6B000D; Thu, 4 Mar 2021 23:19:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BBA7F6B000E; Thu, 4 Mar 2021 23:19:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A335C6B0010; Thu, 4 Mar 2021 23:19:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0092.hostedemail.com [216.40.44.92]) by kanga.kvack.org (Postfix) with ESMTP id 83F9C6B000D for ; Thu, 4 Mar 2021 23:19:33 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4DDDB364F for ; Fri, 5 Mar 2021 04:19:33 +0000 (UTC) X-FDA: 77884516626.20.C40DFDB Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 6BC70C0001EA for ; Fri, 5 Mar 2021 04:19:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=M6kHCP2ErIBVLFqPGPar7eIkGAUP7r+SY6+0x2DcRsk=; b=BtYx3on1gAjsZVKoYT5FUu6v1a EM/Rq722gHomS4jffeqmg1qCOPQ3GOB7fJHzuQhCgBrMckl8DVZEtA+XQTEvlXZInbF1+iRJaT6bg 8JCeAdRuleiE8u0XQtFMkcKSQKQClhQTFCxIEI4kQen8VzaQ+EIkUt8jXUqzew4jEL6NrHfeI1gV1 mM50ZICpfcjvM15fTiI5Td0yb5iK0naAb+HBVoEqWVIJ3EynFXLoT537u+r6BtjTMOiB4Exd4seWS 6/z4b9wZZ+HXyla9cicOdUN/ax8VBu5NCCeYG2aFHOxRpVEKfrbEYSFRpbeAIuJfQ6OwkXKxj6qJk PJnEd3kQ==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1vr-00A3T3-LO; Fri, 05 Mar 2021 04:19:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Zi Yan Subject: [PATCH v4 05/25] mm: Add put_folio Date: Fri, 5 Mar 2021 04:18:41 +0000 Message-Id: <20210305041901.2396498-6-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 8u3s3qnycc6spnis1kj6dw4187dazqn1 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6BC70C0001EA Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614917972-549223 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we know we have a folio, we can call put_folio() instead of put_page() and save the overhead of calling compound_head(). Also skips the devmap checks. This commit looks like it should be a no-op, but actually saves 1714 bytes of text with the distro-derived config that I'm testing. Some functions grow a little while others shrink. I presume the compiler is making different inlining decisions. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan --- include/linux/mm.h | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ecfe202aa4ec..30fd431b1b05 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1205,9 +1205,15 @@ static inline __must_check bool try_get_page(struct page *page) return true; } +static inline void put_folio(struct folio *folio) +{ + if (put_page_testzero(&folio->page)) + __put_page(&folio->page); +} + static inline void put_page(struct page *page) { - page = compound_head(page); + struct folio *folio = page_folio(page); /* * For devmap managed pages we need to catch refcount transition from @@ -1215,13 +1221,12 @@ static inline void put_page(struct page *page) * need to inform the device driver through callback. See * include/linux/memremap.h and HMM for details. */ - if (page_is_devmap_managed(page)) { - put_devmap_managed_page(page); + if (page_is_devmap_managed(&folio->page)) { + put_devmap_managed_page(&folio->page); return; } - if (put_page_testzero(page)) - __put_page(page); + put_folio(folio); } /* From patchwork Fri Mar 5 04:18:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8ADA2C433DB for ; Fri, 5 Mar 2021 04:19:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 410FB64FF0 for ; Fri, 5 Mar 2021 04:19:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 410FB64FF0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CA40C6B000E; Thu, 4 Mar 2021 23:19:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C54346B0010; Thu, 4 Mar 2021 23:19:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF68E6B0012; Thu, 4 Mar 2021 23:19:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id 8DBE46B000E for ; Thu, 4 Mar 2021 23:19:47 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 50893180ACC54 for ; Fri, 5 Mar 2021 04:19:47 +0000 (UTC) X-FDA: 77884517214.21.9243679 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf10.hostedemail.com (Postfix) with ESMTP id 2CAB5407F8E8 for ; Fri, 5 Mar 2021 04:19:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=d50BWWwyptEXh1B6An6n9AJNYi6qLn96OD0apF8yUVA=; b=ALmZiyAevE8ylgfKYrhR8VnqUL QIi7RhfS1Wv9c79tU1+Sda1hU6HUss0VHmRBPZTvQXfdaQXsQ0uTkXWnh+GR3mIdBqgucMiAqRO+S LkF6HBqm6gRgivRU3VGTXzO7R2LDruxqLqgmC/gHrnhNkCTSCUcmqW+gg84DSoVatxqDag/7eYIln zOHhrTLzwwZge5IE4I2XesegOaIDj/eBnR1MsT3USqjDB198Sr+eNOwrFMLQMybPq9ZNSpaUAf5FI P/JqEuT7x6zYrh7OIO3nw42N9lOa2BjBDpocerIyHMjdQ/LO6SO4sXDeCFtuT4Ad+pEio5CF8O7iF V7Ay+uqg==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1vx-00A3TP-GY; Fri, 05 Mar 2021 04:19:26 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Zi Yan Subject: [PATCH v4 06/25] mm: Add get_folio Date: Fri, 5 Mar 2021 04:18:42 +0000 Message-Id: <20210305041901.2396498-7-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 2CAB5407F8E8 X-Stat-Signature: u9zd6jke3ttr44ie78c91c74fr57rfas Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614917985-72646 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we know we have a folio, we can call get_folio() instead of get_page() and save the overhead of calling compound_head(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan --- include/linux/mm.h | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 30fd431b1b05..4595955805f8 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1177,18 +1177,19 @@ static inline bool is_pci_p2pdma_page(const struct page *page) } /* 127: arbitrary random number, small enough to assemble well */ -#define page_ref_zero_or_close_to_overflow(page) \ - ((unsigned int) page_ref_count(page) + 127u <= 127u) +#define folio_ref_zero_or_close_to_overflow(folio) \ + ((unsigned int) page_ref_count(&folio->page) + 127u <= 127u) + +static inline void get_folio(struct folio *folio) +{ + /* Getting a page requires an already elevated page->_refcount. */ + VM_BUG_ON_FOLIO(folio_ref_zero_or_close_to_overflow(folio), folio); + page_ref_inc(&folio->page); +} static inline void get_page(struct page *page) { - page = compound_head(page); - /* - * Getting a normal page or the head of a compound page - * requires to already have an elevated page->_refcount. - */ - VM_BUG_ON_PAGE(page_ref_zero_or_close_to_overflow(page), page); - page_ref_inc(page); + get_folio(page_folio(page)); } bool __must_check try_grab_page(struct page *page, unsigned int flags); From patchwork Fri Mar 5 04:18:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DB04C433E0 for ; Fri, 5 Mar 2021 04:20:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3F99264FE4 for ; Fri, 5 Mar 2021 04:20:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3F99264FE4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CF0976B0010; Thu, 4 Mar 2021 23:20:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CA03E6B0012; Thu, 4 Mar 2021 23:20:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B69116B0022; Thu, 4 Mar 2021 23:20:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id 98F986B0010 for ; Thu, 4 Mar 2021 23:20:42 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6233C364F for ; Fri, 5 Mar 2021 04:20:42 +0000 (UTC) X-FDA: 77884519524.14.B5FCA09 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 84797C0001FA for ; Fri, 5 Mar 2021 04:20:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=dP5UMVOfza3vmGEPtSLp/DyWFDFLrJcWbOeP/yqlT/g=; b=jZ+XY+kY4DkzL8xI9W+ptjKuDC BE6f7v9Dile/BYECrEL77EDUjJToAGJcpex9iNwpqZdJTbUZKT9s16e/o6dfzwRq75RLKn373Yb5P OQUvCoW6CEstjPO9EBcrTtJwgidoihgFdviLyWlBXRtnBuHZswteNy69ffc6QPnrrA8qbUAbUcf3+ rgjY56IS5mg6BSHFA8YENtL69WVzAAtUw1e+bMM3Z8fKph7ftqW20HVWs/K+QhfSPWkmPIo66NRgo LE4DsJ8wSRZEdBpEQ8XKdRYGVEoXmuruQJtrFIFWWh+m1ynnzsYRV/0XUI2PBRUxl46/6WCkPFdj2 xLyQWlBw==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1w5-00A3To-HA; Fri, 05 Mar 2021 04:19:42 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 07/25] mm: Create FolioFlags Date: Fri, 5 Mar 2021 04:18:43 +0000 Message-Id: <20210305041901.2396498-8-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 84797C0001FA X-Stat-Signature: pqfqosa6ebxqouxurwbg9xq3x74kkd4h Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918039-85097 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These new functions are the folio analogues of the PageFlags functions. If CONFIG_DEBUG_VM_PGFLAGS is enabled, we check the folio is not a tail page at every invocation. Note that this will also catch the PagePoisoned case as a poisoned page has every bit set, which would include PageTail. This saves 1740 bytes of text with the distro-derived config that I'm testing due to removing a double call to compound_head() in PageSwapCache(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/fscache.h | 5 ++ include/linux/page-flags.h | 104 ++++++++++++++++++++++++++++++------- 2 files changed, 89 insertions(+), 20 deletions(-) diff --git a/include/linux/fscache.h b/include/linux/fscache.h index a1c928fe98e7..c274006f4037 100644 --- a/include/linux/fscache.h +++ b/include/linux/fscache.h @@ -38,6 +38,11 @@ #define ClearPageFsCache(page) ClearPagePrivate2((page)) #define TestSetPageFsCache(page) TestSetPagePrivate2((page)) #define TestClearPageFsCache(page) TestClearPagePrivate2((page)) +#define FolioFsCache(folio) FolioPrivate2((folio)) +#define SetFolioFsCache(folio) SetFolioPrivate2((folio)) +#define ClearFolioFsCache(folio) ClearFolioPrivate2((folio)) +#define TestSetFolioFsCache(folio) TestSetFolioPrivate2((folio)) +#define TestClearFolioFsCache(folio) TestClearFolioPrivate2((folio)) /* pattern used to fill dead space in an index entry */ #define FSCACHE_INDEX_DEADFILL_PATTERN 0x79 diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 04a34c08e0a6..90381858d901 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -212,6 +212,12 @@ static inline void page_init_poison(struct page *page, size_t size) } #endif +static unsigned long *folio_flags(struct folio *folio) +{ + VM_BUG_ON_PGFLAGS(PageTail(&folio->page), &folio->page); + return &folio->page.flags; +} + /* * Page flags policies wrt compound pages * @@ -260,30 +266,44 @@ static inline void page_init_poison(struct page *page, size_t size) * Macros to create function definitions for page flags */ #define TESTPAGEFLAG(uname, lname, policy) \ +static __always_inline int Folio##uname(struct folio *folio) \ + { return test_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline int Page##uname(struct page *page) \ { return test_bit(PG_##lname, &policy(page, 0)->flags); } #define SETPAGEFLAG(uname, lname, policy) \ +static __always_inline void SetFolio##uname(struct folio *folio) \ + { set_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline void SetPage##uname(struct page *page) \ { set_bit(PG_##lname, &policy(page, 1)->flags); } #define CLEARPAGEFLAG(uname, lname, policy) \ +static __always_inline void ClearFolio##uname(struct folio *folio) \ + { clear_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline void ClearPage##uname(struct page *page) \ { clear_bit(PG_##lname, &policy(page, 1)->flags); } #define __SETPAGEFLAG(uname, lname, policy) \ +static __always_inline void __SetFolio##uname(struct folio *folio) \ + { __set_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline void __SetPage##uname(struct page *page) \ { __set_bit(PG_##lname, &policy(page, 1)->flags); } #define __CLEARPAGEFLAG(uname, lname, policy) \ +static __always_inline void __ClearFolio##uname(struct folio *folio) \ + { __clear_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline void __ClearPage##uname(struct page *page) \ { __clear_bit(PG_##lname, &policy(page, 1)->flags); } #define TESTSETFLAG(uname, lname, policy) \ +static __always_inline int TestSetFolio##uname(struct folio *folio) \ + { return test_and_set_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline int TestSetPage##uname(struct page *page) \ { return test_and_set_bit(PG_##lname, &policy(page, 1)->flags); } #define TESTCLEARFLAG(uname, lname, policy) \ +static __always_inline int TestClearFolio##uname(struct folio *folio) \ + { return test_and_clear_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline int TestClearPage##uname(struct page *page) \ { return test_and_clear_bit(PG_##lname, &policy(page, 1)->flags); } @@ -302,21 +322,27 @@ static __always_inline int TestClearPage##uname(struct page *page) \ TESTCLEARFLAG(uname, lname, policy) #define TESTPAGEFLAG_FALSE(uname) \ +static inline int Folio##uname(const struct folio *folio) { return 0; } \ static inline int Page##uname(const struct page *page) { return 0; } #define SETPAGEFLAG_NOOP(uname) \ +static inline void SetFolio##uname(struct folio *folio) { } \ static inline void SetPage##uname(struct page *page) { } #define CLEARPAGEFLAG_NOOP(uname) \ +static inline void ClearFolio##uname(struct folio *folio) { } \ static inline void ClearPage##uname(struct page *page) { } #define __CLEARPAGEFLAG_NOOP(uname) \ +static inline void __ClearFolio##uname(struct folio *folio) { } \ static inline void __ClearPage##uname(struct page *page) { } #define TESTSETFLAG_FALSE(uname) \ +static inline int TestSetFolio##uname(struct folio *folio) { return 0; } \ static inline int TestSetPage##uname(struct page *page) { return 0; } #define TESTCLEARFLAG_FALSE(uname) \ +static inline int TestClearFolio##uname(struct folio *folio) { return 0; } \ static inline int TestClearPage##uname(struct page *page) { return 0; } #define PAGEFLAG_FALSE(uname) TESTPAGEFLAG_FALSE(uname) \ @@ -393,14 +419,18 @@ PAGEFLAG_FALSE(HighMem) #endif #ifdef CONFIG_SWAP -static __always_inline int PageSwapCache(struct page *page) +static __always_inline bool FolioSwapCache(struct folio *folio) { -#ifdef CONFIG_THP_SWAP - page = compound_head(page); -#endif - return PageSwapBacked(page) && test_bit(PG_swapcache, &page->flags); + return FolioSwapBacked(folio) && + test_bit(PG_swapcache, folio_flags(folio)); } + +static __always_inline bool PageSwapCache(struct page *page) +{ + return FolioSwapCache(page_folio(page)); +} + SETPAGEFLAG(SwapCache, swapcache, PF_NO_TAIL) CLEARPAGEFLAG(SwapCache, swapcache, PF_NO_TAIL) #else @@ -478,10 +508,14 @@ static __always_inline int PageMappingFlags(struct page *page) return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) != 0; } -static __always_inline int PageAnon(struct page *page) +static __always_inline bool FolioAnon(struct folio *folio) { - page = compound_head(page); - return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0; + return ((unsigned long)folio->page.mapping & PAGE_MAPPING_ANON) != 0; +} + +static __always_inline bool PageAnon(struct page *page) +{ + return FolioAnon(page_folio(page)); } static __always_inline int __PageMovable(struct page *page) @@ -509,18 +543,16 @@ TESTPAGEFLAG_FALSE(Ksm) u64 stable_page_flags(struct page *page); -static inline int PageUptodate(struct page *page) +static inline int FolioUptodate(struct folio *folio) { - int ret; - page = compound_head(page); - ret = test_bit(PG_uptodate, &(page)->flags); + int ret = test_bit(PG_uptodate, folio_flags(folio)); /* * Must ensure that the data we read out of the page is loaded * _after_ we've loaded page->flags to check for PageUptodate. * We can skip the barrier if the page is not uptodate, because * we wouldn't be reading anything from it. * - * See SetPageUptodate() for the other side of the story. + * See SetFolioUptodate() for the other side of the story. */ if (ret) smp_rmb(); @@ -528,23 +560,36 @@ static inline int PageUptodate(struct page *page) return ret; } -static __always_inline void __SetPageUptodate(struct page *page) +static inline int PageUptodate(struct page *page) +{ + return FolioUptodate(page_folio(page)); +} + +static __always_inline void __SetFolioUptodate(struct folio *folio) { - VM_BUG_ON_PAGE(PageTail(page), page); smp_wmb(); - __set_bit(PG_uptodate, &page->flags); + __set_bit(PG_uptodate, folio_flags(folio)); } -static __always_inline void SetPageUptodate(struct page *page) +static __always_inline void SetFolioUptodate(struct folio *folio) { - VM_BUG_ON_PAGE(PageTail(page), page); /* * Memory barrier must be issued before setting the PG_uptodate bit, * so that all previous stores issued in order to bring the page * uptodate are actually visible before PageUptodate becomes true. */ smp_wmb(); - set_bit(PG_uptodate, &page->flags); + set_bit(PG_uptodate, folio_flags(folio)); +} + +static __always_inline void __SetPageUptodate(struct page *page) +{ + __SetFolioUptodate((struct folio *)page); +} + +static __always_inline void SetPageUptodate(struct page *page) +{ + SetFolioUptodate((struct folio *)page); } CLEARPAGEFLAG(Uptodate, uptodate, PF_NO_TAIL) @@ -569,6 +614,17 @@ static inline void set_page_writeback_keepwrite(struct page *page) __PAGEFLAG(Head, head, PF_ANY) CLEARPAGEFLAG(Head, head, PF_ANY) +/* Whether there are one or multiple pages in a folio */ +static inline bool FolioSingle(struct folio *folio) +{ + return !FolioHead(folio); +} + +static inline bool FolioMulti(struct folio *folio) +{ + return FolioHead(folio); +} + static __always_inline void set_compound_head(struct page *page, struct page *head) { WRITE_ONCE(page->compound_head, (unsigned long)head + 1); @@ -592,12 +648,15 @@ static inline void ClearPageCompound(struct page *page) #ifdef CONFIG_HUGETLB_PAGE int PageHuge(struct page *page); int PageHeadHuge(struct page *page); +static inline bool FolioHuge(struct folio *folio) +{ + return PageHeadHuge(&folio->page); +} #else TESTPAGEFLAG_FALSE(Huge) TESTPAGEFLAG_FALSE(HeadHuge) #endif - #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* * PageHuge() only returns true for hugetlbfs pages, but not for @@ -844,6 +903,11 @@ static inline int page_has_private(struct page *page) return !!(page->flags & PAGE_FLAGS_PRIVATE); } +static inline bool folio_has_private(struct folio *folio) +{ + return page_has_private(&folio->page); +} + #undef PF_ANY #undef PF_HEAD #undef PF_ONLY_HEAD From patchwork Fri Mar 5 04:18:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7C05C433E0 for ; Fri, 5 Mar 2021 04:21:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7357965009 for ; Fri, 5 Mar 2021 04:21:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7357965009 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 108706B0007; Thu, 4 Mar 2021 23:21:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E1216B0008; Thu, 4 Mar 2021 23:21:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC3B26B000A; Thu, 4 Mar 2021 23:21:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0097.hostedemail.com [216.40.44.97]) by kanga.kvack.org (Postfix) with ESMTP id CD55E6B0007 for ; Thu, 4 Mar 2021 23:21:28 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 8BED1181AEF10 for ; Fri, 5 Mar 2021 04:21:28 +0000 (UTC) X-FDA: 77884521456.04.2AA98F5 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf17.hostedemail.com (Postfix) with ESMTP id D889A40001DE for ; Fri, 5 Mar 2021 04:21:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=VTFfmSh3PtwZWFi9FZ9t/kJrbRPQg2Phw+OnoZy+bTo=; b=FeDX+TJ7yiCS7D3DPp2FWORI0Z 67MDf1+YHsLWiW14UZeVUfTITA8le6UqpwTPq2/duAAfwTNk97c+S4beIQfjJXCr9E2QE0Qoymwu0 ITKHEC9t7YRRWalsjC6niGbQXqk5rhE4Lk4E7bonvtdOvQNhfoYd1kKsmWogUkZyTdLt3SwBtzdSr tSbY8b2s9vrciwG8zhE1zmJVe/SxGvPbKDoT6yYCLd+a5P64/halNz5uJ0xirA7oMPov8WUBASs40 KQh7agI9n0CTMb+cCkKOstq9SYUxzBaAinloxaLESoyyQXHDyFykyib+DCZdSwfEvHflIzYxv4hCy /q7H9jEQ==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1wg-00A3Zj-Tw; Fri, 05 Mar 2021 04:20:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 08/25] mm: Handle per-folio private data Date: Fri, 5 Mar 2021 04:18:44 +0000 Message-Id: <20210305041901.2396498-9-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D889A40001DE X-Stat-Signature: ibkzbzecwhds45jzzianed7m9t346hfn Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf17; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918087-890358 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add folio_private() and set_folio_private() which mirror page_private() and set_page_private() -- ie folio private data is the same as page private data. Turn attach_page_private() into attach_folio_private() and reimplement attach_page_private() as a wrapper. No filesystem which uses page private data currently supports compound pages, so we're free to define the rules. attach_page_private() may only be called on a head page; if you want to add private data to a tail page, you can call set_page_private() directly (and shouldn't increment the page refcount! That should be done when adding private data to the head page / folio). This saves 886 bytes of text with the distro-derived config that I'm testing due to removing the calls to compound_head() in get_page() & put_page(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm_types.h | 16 ++++++++++++++ include/linux/pagemap.h | 48 ++++++++++++++++++++++++---------------- 2 files changed, 45 insertions(+), 19 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index a311cb48526f..b650423fcba6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -258,6 +258,12 @@ static inline atomic_t *compound_pincount_ptr(struct page *page) #define PAGE_FRAG_CACHE_MAX_SIZE __ALIGN_MASK(32768, ~PAGE_MASK) #define PAGE_FRAG_CACHE_MAX_ORDER get_order(PAGE_FRAG_CACHE_MAX_SIZE) +/* + * page_private can be used on tail pages. However, PagePrivate is only + * checked by the VM on the head page. So page_private on the tail pages + * should be used for data that's ancillary to the head page (eg attaching + * buffer heads to tail pages after attaching buffer heads to the head page) + */ #define page_private(page) ((page)->private) static inline void set_page_private(struct page *page, unsigned long private) @@ -265,6 +271,16 @@ static inline void set_page_private(struct page *page, unsigned long private) page->private = private; } +static inline unsigned long folio_private(struct folio *folio) +{ + return folio->page.private; +} + +static inline void set_folio_private(struct folio *folio, unsigned long v) +{ + folio->page.private = v; +} + struct page_frag_cache { void * va; #if (PAGE_SIZE < PAGE_FRAG_CACHE_MAX_SIZE) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 20225b067583..f07c03da83f6 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -245,42 +245,52 @@ static inline int page_cache_add_speculative(struct page *page, int count) } /** - * attach_page_private - Attach private data to a page. - * @page: Page to attach data to. - * @data: Data to attach to page. + * attach_folio_private - Attach private data to a folio. + * @folio: Folio to attach data to. + * @data: Data to attach to folio. * - * Attaching private data to a page increments the page's reference count. - * The data must be detached before the page will be freed. + * Attaching private data to a folio increments the page's reference count. + * The data must be detached before the folio will be freed. */ -static inline void attach_page_private(struct page *page, void *data) +static inline void attach_folio_private(struct folio *folio, void *data) { - get_page(page); - set_page_private(page, (unsigned long)data); - SetPagePrivate(page); + get_folio(folio); + set_folio_private(folio, (unsigned long)data); + SetFolioPrivate(folio); } /** - * detach_page_private - Detach private data from a page. - * @page: Page to detach data from. + * detach_folio_private - Detach private data from a folio. + * @folio: Folio to detach data from. * - * Removes the data that was previously attached to the page and decrements + * Removes the data that was previously attached to the folio and decrements * the refcount on the page. * - * Return: Data that was attached to the page. + * Return: Data that was attached to the folio. */ -static inline void *detach_page_private(struct page *page) +static inline void *detach_folio_private(struct folio *folio) { - void *data = (void *)page_private(page); + void *data = (void *)folio_private(folio); - if (!PagePrivate(page)) + if (!FolioPrivate(folio)) return NULL; - ClearPagePrivate(page); - set_page_private(page, 0); - put_page(page); + ClearFolioPrivate(folio); + set_folio_private(folio, 0); + put_folio(folio); return data; } +static inline void attach_page_private(struct page *page, void *data) +{ + attach_folio_private((struct folio *)page, data); +} + +static inline void *detach_page_private(struct page *page) +{ + return detach_folio_private((struct folio *)page); +} + #ifdef CONFIG_NUMA extern struct page *__page_cache_alloc(gfp_t gfp); #else From patchwork Fri Mar 5 04:18:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEB03C433E0 for ; Fri, 5 Mar 2021 04:22:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7505165016 for ; Fri, 5 Mar 2021 04:22:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7505165016 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0E0EA6B000A; Thu, 4 Mar 2021 23:22:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B91C6B000C; Thu, 4 Mar 2021 23:22:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC33F6B000E; Thu, 4 Mar 2021 23:22:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0215.hostedemail.com [216.40.44.215]) by kanga.kvack.org (Postfix) with ESMTP id D015F6B000A for ; Thu, 4 Mar 2021 23:22:13 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 994E1180ACC44 for ; Fri, 5 Mar 2021 04:22:13 +0000 (UTC) X-FDA: 77884523346.07.3B79766 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id E19BEC0001EE for ; Fri, 5 Mar 2021 04:22:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=VD0Fvg6cmd14tiXkK8z0qmCk4W9zwnqQgUrYNW/arts=; b=JpUjN9GY8ZgJniUfRJZmlBURQ2 nIgf3UKyig4Swhl+XLzue7r1CE7cw/LigQZDDdEXQPrgenhmeC15rneYkVY4QPkvV18CaSDbox2bX 7RxVqgp8CLAY+9zduAnD5qo1j1OBj30DLKOiZVSIgpDa9OsIS0WPmwZtMZVSAwsIsNuQU6tMqkhQu 2mAec1BSqPl4qVSd3GTHhdWYoTzgI/xGdofk9SgsAVNRcAM4QBeQb4PoB+8R7/wZNbbAxFSaV4psK bCH2eSemJ3JmZelTPpYNbt0Tm06aBUWRSKEXQsYcDr/xkC8mgpbPfZdHn2ljUDfmi/AFb8pBvSDky v0fDfY2w==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1xU-00A3cR-Fh; Fri, 05 Mar 2021 04:21:13 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 09/25] mm: Add folio_index, folio_page and folio_contains Date: Fri, 5 Mar 2021 04:18:45 +0000 Message-Id: <20210305041901.2396498-10-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: E19BEC0001EE X-Stat-Signature: np3p9hujzhjfxzgqdhpgnnqpttrdgw1e Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918130-542317 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: folio_index() is the equivalent of page_index() for folios. folio_page() finds the page in a folio for a page cache index. folio_contains() tells you whether a folio contains a particular page cache index. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index f07c03da83f6..5094b50f7680 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -447,6 +447,29 @@ static inline bool thp_contains(struct page *head, pgoff_t index) return page_index(head) == (index & ~(thp_nr_pages(head) - 1UL)); } +static inline pgoff_t folio_index(struct folio *folio) +{ + if (unlikely(FolioSwapCache(folio))) + return __page_file_index(&folio->page); + return folio->page.index; +} + +static inline struct page *folio_page(struct folio *folio, pgoff_t index) +{ + index -= folio_index(folio); + VM_BUG_ON_FOLIO(index >= folio_nr_pages(folio), folio); + return &folio->page + index; +} + +/* Does this folio contain this index? */ +static inline bool folio_contains(struct folio *folio, pgoff_t index) +{ + /* HugeTLBfs indexes the page cache in units of hpage_size */ + if (PageHuge(&folio->page)) + return folio->page.index == index; + return index - folio_index(folio) < folio_nr_pages(folio); +} + /* * Given the page we found in the page cache, return the page corresponding * to this index in the file From patchwork Fri Mar 5 04:18:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DAAAC433DB for ; Fri, 5 Mar 2021 04:22:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D85766500C for ; Fri, 5 Mar 2021 04:22:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D85766500C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F94D6B0007; Thu, 4 Mar 2021 23:22:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6D0C96B000C; Thu, 4 Mar 2021 23:22:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5728A6B000D; Thu, 4 Mar 2021 23:22:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id 3EAAC6B0007 for ; Thu, 4 Mar 2021 23:22:52 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 05FA98249980 for ; Fri, 5 Mar 2021 04:22:52 +0000 (UTC) X-FDA: 77884524984.09.C2DAE4E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 7652C132 for ; Fri, 5 Mar 2021 04:22:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=PJMDVzBGNFdjhf6t/3/5VE6jTBhydf1jdb3Lpx2OhtA=; b=JTROAvijqYfvnr+BFahul4HXq3 EsnBBVkU4zMVXGycqIDB1bN6gcx+NFJcCRG2tFUqDOUIGo9JzySGOAvIfGwvHTS9P5BKRYfzrr8L7 93QGCGwRvlxLquF2WmwTsXBdnWcA5523Bg0S242U4kX3YCiH53YETO/F5jlpu7dahaXjdO74twtDX vn92MZW+u0J0LAgZjM1yW176QThhrmWi3cTouEs+431OotR76T+VvF2ZWqC7pAZ5++zIArrqwIOVI M9YWGJVjbKB1eJPMqSIg52bnOnxolawt+NZqjIOsY+YCBKMvY4DaHa/PUfmlbijIR/WAYKnuKhUg4 46nVAI2w==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1y1-00A3fp-Al; Fri, 05 Mar 2021 04:21:50 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 10/25] mm/util: Add folio_mapping and folio_file_mapping Date: Fri, 5 Mar 2021 04:18:46 +0000 Message-Id: <20210305041901.2396498-11-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: ky4qt9bo6ssqem7pge8bgh3hegf85mi1 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7652C132 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918170-118816 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These are the folio equivalent of page_mapping() and page_file_mapping(). Add an out-of-line page_mapping() wrapper around folio_mapping() in order to prevent the page_folio() call from bloating every caller of page_mapping(). Adjust page_file_mapping() and page_mapping_file() to use folios internally. This ends up saving 102 bytes of text overall. folio_mapping() is 45 bytes shorter than page_mapping() was, but the compiler chooses to inline folio_mapping() into page_mapping_file(), for a net increase of 69 bytes in the core. This is made up for by a few bytes less in dozens of nfs functions (which call page_file_mapping()). The small amount of difference appears to be a slight change in gcc's register allocation decisions, which allow: 48 8b 56 08 mov 0x8(%rsi),%rdx 48 8d 42 ff lea -0x1(%rdx),%rax 83 e2 01 and $0x1,%edx 48 0f 44 c6 cmove %rsi,%rax to become: 48 8b 46 08 mov 0x8(%rsi),%rax 48 8d 78 ff lea -0x1(%rax),%rdi a8 01 test $0x1,%al 48 0f 44 fe cmove %rsi,%rdi Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 14 -------------- include/linux/pagemap.h | 17 +++++++++++++++++ mm/Makefile | 2 +- mm/folio-compat.c | 13 +++++++++++++ mm/swapfile.c | 6 +++--- mm/util.c | 20 ++++++++++---------- 6 files changed, 44 insertions(+), 28 deletions(-) create mode 100644 mm/folio-compat.c diff --git a/include/linux/mm.h b/include/linux/mm.h index 4595955805f8..9199b59ee8da 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1603,19 +1603,6 @@ void page_address_init(void); extern void *page_rmapping(struct page *page); extern struct anon_vma *page_anon_vma(struct page *page); -extern struct address_space *page_mapping(struct page *page); - -extern struct address_space *__page_file_mapping(struct page *); - -static inline -struct address_space *page_file_mapping(struct page *page) -{ - if (unlikely(PageSwapCache(page))) - return __page_file_mapping(page); - - return page->mapping; -} - extern pgoff_t __page_file_index(struct page *page); /* @@ -1630,7 +1617,6 @@ static inline pgoff_t page_index(struct page *page) } bool page_mapped(struct page *page); -struct address_space *page_mapping(struct page *page); struct address_space *page_mapping_file(struct page *page); /* diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 5094b50f7680..5a2c0764d7c0 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -157,6 +157,23 @@ static inline void filemap_nr_thps_dec(struct address_space *mapping) void release_pages(struct page **pages, int nr); +struct address_space *page_mapping(struct page *); +struct address_space *folio_mapping(struct folio *); +struct address_space *__folio_file_mapping(struct folio *); + +static inline struct address_space *folio_file_mapping(struct folio *folio) +{ + if (unlikely(FolioSwapCache(folio))) + return __folio_file_mapping(folio); + + return folio->page.mapping; +} + +static inline struct address_space *page_file_mapping(struct page *page) +{ + return folio_file_mapping(page_folio(page)); +} + /* * speculatively take a reference to a page. * If the page is free (_refcount == 0), then _refcount is untouched, and 0 diff --git a/mm/Makefile b/mm/Makefile index 72227b24a616..ceb0089efd29 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -46,7 +46,7 @@ mmu-$(CONFIG_MMU) += process_vm_access.o endif obj-y := filemap.o mempool.o oom_kill.o fadvise.o \ - maccess.o page-writeback.o \ + maccess.o page-writeback.o folio-compat.o \ readahead.o swap.o truncate.o vmscan.o shmem.o \ util.o mmzone.o vmstat.o backing-dev.o \ mm_init.o percpu.o slab_common.o \ diff --git a/mm/folio-compat.c b/mm/folio-compat.c new file mode 100644 index 000000000000..5e107aa30a62 --- /dev/null +++ b/mm/folio-compat.c @@ -0,0 +1,13 @@ +/* + * Compatibility functions which bloat the callers too much to make inline. + * All of the callers of these functions should be converted to use folios + * eventually. + */ + +#include + +struct address_space *page_mapping(struct page *page) +{ + return folio_mapping(page_folio(page)); +} +EXPORT_SYMBOL(page_mapping); diff --git a/mm/swapfile.c b/mm/swapfile.c index 084a5b9a18e5..b80af7537d9e 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3535,11 +3535,11 @@ struct swap_info_struct *page_swap_info(struct page *page) /* * out-of-line __page_file_ methods to avoid include hell. */ -struct address_space *__page_file_mapping(struct page *page) +struct address_space *__folio_file_mapping(struct folio *folio) { - return page_swap_info(page)->swap_file->f_mapping; + return page_swap_info(&folio->page)->swap_file->f_mapping; } -EXPORT_SYMBOL_GPL(__page_file_mapping); +EXPORT_SYMBOL_GPL(__folio_file_mapping); pgoff_t __page_file_index(struct page *page) { diff --git a/mm/util.c b/mm/util.c index 54870226cea6..9ab72cfa4aa1 100644 --- a/mm/util.c +++ b/mm/util.c @@ -686,39 +686,39 @@ struct anon_vma *page_anon_vma(struct page *page) return __page_rmapping(page); } -struct address_space *page_mapping(struct page *page) +struct address_space *folio_mapping(struct folio *folio) { struct address_space *mapping; - page = compound_head(page); - /* This happens if someone calls flush_dcache_page on slab page */ - if (unlikely(PageSlab(page))) + if (unlikely(FolioSlab(folio))) return NULL; - if (unlikely(PageSwapCache(page))) { + if (unlikely(FolioSwapCache(folio))) { swp_entry_t entry; - entry.val = page_private(page); + entry.val = folio_private(folio); return swap_address_space(entry); } - mapping = page->mapping; + mapping = folio->page.mapping; if ((unsigned long)mapping & PAGE_MAPPING_ANON) return NULL; return (void *)((unsigned long)mapping & ~PAGE_MAPPING_FLAGS); } -EXPORT_SYMBOL(page_mapping); +EXPORT_SYMBOL(folio_mapping); /* * For file cache pages, return the address_space, otherwise return NULL */ struct address_space *page_mapping_file(struct page *page) { - if (unlikely(PageSwapCache(page))) + struct folio *folio = page_folio(page); + + if (unlikely(FolioSwapCache(folio))) return NULL; - return page_mapping(page); + return folio_mapping(folio); } /* Slow path of page_mapcount() for compound pages */ From patchwork Fri Mar 5 04:18:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17599C433DB for ; Fri, 5 Mar 2021 04:23:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A993C65014 for ; Fri, 5 Mar 2021 04:23:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A993C65014 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 439476B0008; Thu, 4 Mar 2021 23:23:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E8926B000A; Thu, 4 Mar 2021 23:23:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D8246B000D; Thu, 4 Mar 2021 23:23:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0212.hostedemail.com [216.40.44.212]) by kanga.kvack.org (Postfix) with ESMTP id 0CA756B0008 for ; Thu, 4 Mar 2021 23:23:25 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CA9EA3658 for ; Fri, 5 Mar 2021 04:23:24 +0000 (UTC) X-FDA: 77884526328.02.AF4364B Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP id CF5D1407F8F3 for ; Fri, 5 Mar 2021 04:23:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=OeuwNxzVk52kWY8FtpOwc6yIjdYdkVvfN03G7e/qaqE=; b=F8xZYKPdFD5KN2VsEC1HcYRhhR ja3lBgAfbAoA0uq72SIm73VsH2GoLB3JKjH4mIQD4TwyYcoUZUFMR4cetnM/J22qvf/CH8bzj6Vw9 uAxTMANRs47KrtVzL63x7NhKoRYjCaQoYClE2xoL/3/9U9nP50zK8w7v8Kz08d19PwrgDoc5aZ5KV droHRWOOqMvgGxhuA0aU7mIqlkYM21pvUlxPnOV8f1Gjf+8ruDHzSlmr9Dzx8FwjyysJUNThaxjrc +xBaz832GU4n63hF/x3pY1vz4vhrR25O929eafuB7yLOAjTX/cxKCB6zGkuz+RglmxfDv0i+w2qdt yI+KO2XA==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1ya-00A3i5-2g; Fri, 05 Mar 2021 04:22:16 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 11/25] mm/memcg: Add folio wrappers for various memcontrol functions Date: Fri, 5 Mar 2021 04:18:47 +0000 Message-Id: <20210305041901.2396498-12-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: bt5jfyykd5io1k6yu39ax4ppxa11gbtp X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: CF5D1407F8F3 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918200-840575 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add new functions folio_memcg(), lock_folio_memcg(), unlock_folio_memcg() and mem_cgroup_folio_lruvec(). These will all be used by other functions later. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e6dc793d587d..564ce61319d4 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -383,6 +383,11 @@ static inline struct mem_cgroup *page_memcg(struct page *page) return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } +static inline struct mem_cgroup *folio_memcg(struct folio *folio) +{ + return page_memcg(&folio->page); +} + /* * page_memcg_rcu - locklessly get the memory cgroup associated with a page * @page: a pointer to the page struct @@ -871,6 +876,16 @@ struct mem_cgroup *lock_page_memcg(struct page *page); void __unlock_page_memcg(struct mem_cgroup *memcg); void unlock_page_memcg(struct page *page); +static inline struct mem_cgroup *lock_folio_memcg(struct folio *folio) +{ + return lock_page_memcg(&folio->page); +} + +static inline void unlock_folio_memcg(struct folio *folio) +{ + unlock_page_memcg(&folio->page); +} + /* * idx can be of type enum memcg_stat_item or node_stat_item. * Keep in sync with memcg_exact_page_state(). @@ -1291,6 +1306,11 @@ mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg) { } +static inline struct mem_cgroup *lock_folio_memcg(struct folio *folio) +{ + return NULL; +} + static inline struct mem_cgroup *lock_page_memcg(struct page *page) { return NULL; @@ -1300,6 +1320,10 @@ static inline void __unlock_page_memcg(struct mem_cgroup *memcg) { } +static inline void unlock_folio_memcg(struct folio *folio) +{ +} + static inline void unlock_page_memcg(struct page *page) { } @@ -1431,6 +1455,12 @@ static inline void lruvec_memcg_debug(struct lruvec *lruvec, struct page *page) } #endif /* CONFIG_MEMCG */ +static inline struct lruvec *mem_cgroup_folio_lruvec(struct folio *folio, + struct pglist_data *pgdat) +{ + return mem_cgroup_page_lruvec(&folio->page, pgdat); +} + static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) { __mod_lruvec_kmem_state(p, idx, 1); From patchwork Fri Mar 5 04:18:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0719AC433E6 for ; Fri, 5 Mar 2021 04:23:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9893C65009 for ; Fri, 5 Mar 2021 04:23:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9893C65009 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 66E586B000A; Thu, 4 Mar 2021 23:23:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 522BC6B000E; Thu, 4 Mar 2021 23:23:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34C1D6B000C; Thu, 4 Mar 2021 23:23:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0074.hostedemail.com [216.40.44.74]) by kanga.kvack.org (Postfix) with ESMTP id 12AD66B000A for ; Thu, 4 Mar 2021 23:23:25 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D10333659 for ; Fri, 5 Mar 2021 04:23:24 +0000 (UTC) X-FDA: 77884526328.17.448A969 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id ED65A90009EA for ; Fri, 5 Mar 2021 04:23:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xIH816+bNX6rj//4dyFCa/3R0nIXhyQvmvogKX/pEww=; b=RzZR6JKU2IwuJwPbwsf2uVg5Lh Wp3Q6+TQRqgb0xsZp8tJ1qFYRUT2HcGGMqXG9eRAcbzBXDyvpBA22uZnlQEvSEkdhtbBzYNMIYGHS xpUG6po/ARRkltBJiaFuaeeEIou3ChFiI64/Iy4Qd9A4g5P5CsYEHHDLe/ZW//yoEFHgp8oecVt1k jsK9+Qw7/at9gV062+QlqfHcnysF/jHWEy/do2CpVa93E6vfyURIMqjW6R8AcFZ+u3BMlzYcj8dxp nuIPaxdqBdofF5uDypusr6V02sFOx8fnpOBZ4UyUcYMQ4TeeAgQPh0cV8IAj+PI/Ypeu7Uymd5WSC 0OUF2ubA==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1zE-00A3kQ-3h; Fri, 05 Mar 2021 04:22:50 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 12/25] mm/filemap: Add unlock_folio Date: Fri, 5 Mar 2021 04:18:48 +0000 Message-Id: <20210305041901.2396498-13-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: gd7ufe8bf6db98t8ydp64sdi9cdnqipx X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: ED65A90009EA Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918203-615045 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert unlock_page() to call unlock_folio(). By using a folio we avoid a call to compound_head(). This shortens the function from 39 bytes to 25 and removes 4 instructions on x86-64. Because we still have unlock_page(), it's a net increase of 24 bytes of text. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 3 ++- mm/filemap.c | 27 ++++++++++----------------- mm/folio-compat.c | 6 ++++++ 3 files changed, 18 insertions(+), 18 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 5a2c0764d7c0..a34cf531c100 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -640,7 +640,8 @@ extern int __lock_page_killable(struct page *page); extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags); -extern void unlock_page(struct page *page); +void unlock_page(struct page *page); +void unlock_folio(struct folio *folio); /* * Return true if the page was successfully locked diff --git a/mm/filemap.c b/mm/filemap.c index 3d1635d3be3e..9960ef1b2758 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1408,29 +1408,22 @@ static inline bool clear_bit_unlock_is_negative_byte(long nr, volatile void *mem #endif /** - * unlock_page - unlock a locked page - * @page: the page + * unlock_folio - Unlock a locked folio. + * @folio: The folio. * - * Unlocks the page and wakes up sleepers in wait_on_page_locked(). - * Also wakes sleepers in wait_on_page_writeback() because the wakeup - * mechanism between PageLocked pages and PageWriteback pages is shared. - * But that's OK - sleepers in wait_on_page_writeback() just go back to sleep. + * Unlocks the folio and wakes up any thread sleeping on the page lock. * - * Note that this depends on PG_waiters being the sign bit in the byte - * that contains PG_locked - thus the BUILD_BUG_ON(). That allows us to - * clear the PG_locked bit and test PG_waiters at the same time fairly - * portably (architectures that do LL/SC can test any bit, while x86 can - * test the sign bit). + * Context: May be called from interrupt or process context. May not be + * called from NMI context. */ -void unlock_page(struct page *page) +void unlock_folio(struct folio *folio) { BUILD_BUG_ON(PG_waiters != 7); - page = compound_head(page); - VM_BUG_ON_PAGE(!PageLocked(page), page); - if (clear_bit_unlock_is_negative_byte(PG_locked, &page->flags)) - wake_up_page_bit(page, PG_locked); + VM_BUG_ON_FOLIO(!FolioLocked(folio), folio); + if (clear_bit_unlock_is_negative_byte(PG_locked, folio_flags(folio))) + wake_up_page_bit(&folio->page, PG_locked); } -EXPORT_SYMBOL(unlock_page); +EXPORT_SYMBOL(unlock_folio); /** * end_page_writeback - end writeback against a page diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 5e107aa30a62..02798abf19a1 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -11,3 +11,9 @@ struct address_space *page_mapping(struct page *page) return folio_mapping(page_folio(page)); } EXPORT_SYMBOL(page_mapping); + +void unlock_page(struct page *page) +{ + return unlock_folio(page_folio(page)); +} +EXPORT_SYMBOL(unlock_page); From patchwork Fri Mar 5 04:18:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC5DDC433E0 for ; Fri, 5 Mar 2021 04:23:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6F4B865009 for ; Fri, 5 Mar 2021 04:23:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6F4B865009 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 12A466B000C; Thu, 4 Mar 2021 23:23:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DB736B000D; Thu, 4 Mar 2021 23:23:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE5EE6B000E; Thu, 4 Mar 2021 23:23:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0231.hostedemail.com [216.40.44.231]) by kanga.kvack.org (Postfix) with ESMTP id D2FF86B000C for ; Thu, 4 Mar 2021 23:23:45 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 8F970180ACC44 for ; Fri, 5 Mar 2021 04:23:45 +0000 (UTC) X-FDA: 77884527210.28.769B885 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 13251D6 for ; Fri, 5 Mar 2021 04:23:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=GNVeNTjh8qSM6taS0KFQZnAh6GuTzP3Y17/7pIz0/00=; b=TtnXDCNWldliiyBuYtu5pgwJ3J 6A4Fr7benmulwG2DvO+Q+TKu17CVi0sK/di9mGSFp7zNPuuqFljH4d3J8+ZDFlHh+lbdjSnclzTmq m2AG0Rk0dlKX7oCxQ2HK0vxk/RP8JBUCii22UFO7oj1lXJqt45ezcpAvYWrc9C7ITWGwLfqvzG3dd lNCQuVgZk/RMWV0WxlkClhXYZBi2EDZ7fTXSbAuOiF7rDXtND/aEoSpSWqETFl142UKZ3y1yY/Tf4 QVjIp8iR0Fcx+smd6E2jTMAKnU+0N8cDm2VTu/+NXtO1WKNcIR1xTvHqGKi/14MMh3uZrPEFZjBkR Ec+q3odQ==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1zM-00A3lz-I3; Fri, 05 Mar 2021 04:23:01 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 13/25] mm/filemap: Add lock_folio Date: Fri, 5 Mar 2021 04:18:49 +0000 Message-Id: <20210305041901.2396498-14-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 13251D6 X-Stat-Signature: w6a5m8hz6y1nanmtcur8xxoqq81r1zgb Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918223-590709 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is like lock_page() but for use by callers who know they have a folio. Convert __lock_page() to be __lock_folio(). This saves one call to compound_head() per contended call to lock_page(). Saves 362 bytes of text; mostly from improved register allocation and inlining decisions. __lock_folio is 59 bytes while __lock_page was 79. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 24 +++++++++++++++++++----- mm/filemap.c | 29 +++++++++++++++-------------- 2 files changed, 34 insertions(+), 19 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index a34cf531c100..034e41256340 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -635,7 +635,7 @@ static inline bool wake_page_match(struct wait_page_queue *wait_page, return true; } -extern void __lock_page(struct page *page); +void __lock_folio(struct folio *folio); extern int __lock_page_killable(struct page *page); extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, @@ -643,13 +643,24 @@ extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, void unlock_page(struct page *page); void unlock_folio(struct folio *folio); +static inline bool trylock_folio(struct folio *folio) +{ + return likely(!test_and_set_bit_lock(PG_locked, folio_flags(folio))); +} + /* * Return true if the page was successfully locked */ static inline int trylock_page(struct page *page) { - page = compound_head(page); - return (likely(!test_and_set_bit_lock(PG_locked, &page->flags))); + return trylock_folio(page_folio(page)); +} + +static inline void lock_folio(struct folio *folio) +{ + might_sleep(); + if (!trylock_folio(folio)) + __lock_folio(folio); } /* @@ -657,9 +668,12 @@ static inline int trylock_page(struct page *page) */ static inline void lock_page(struct page *page) { + struct folio *folio; might_sleep(); - if (!trylock_page(page)) - __lock_page(page); + + folio = page_folio(page); + if (!trylock_folio(folio)) + __lock_folio(folio); } /* diff --git a/mm/filemap.c b/mm/filemap.c index 9960ef1b2758..3e3e3c666b94 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1160,7 +1160,7 @@ static void wake_up_page(struct page *page, int bit) */ enum behavior { EXCLUSIVE, /* Hold ref to page and take the bit when woken, like - * __lock_page() waiting on then setting PG_locked. + * __lock_folio() waiting on then setting PG_locked. */ SHARED, /* Hold ref to page and check the bit when woken, like * wait_on_page_writeback() waiting on PG_writeback. @@ -1488,17 +1488,16 @@ void page_endio(struct page *page, bool is_write, int err) EXPORT_SYMBOL_GPL(page_endio); /** - * __lock_page - get a lock on the page, assuming we need to sleep to get it - * @__page: the page to lock + * __lock_folio - Get a lock on the folio, assuming we need to sleep to get it. + * @folio: The folio to lock */ -void __lock_page(struct page *__page) +void __lock_folio(struct folio *folio) { - struct page *page = compound_head(__page); - wait_queue_head_t *q = page_waitqueue(page); - wait_on_page_bit_common(q, page, PG_locked, TASK_UNINTERRUPTIBLE, + wait_queue_head_t *q = page_waitqueue(&folio->page); + wait_on_page_bit_common(q, &folio->page, PG_locked, TASK_UNINTERRUPTIBLE, EXCLUSIVE); } -EXPORT_SYMBOL(__lock_page); +EXPORT_SYMBOL(__lock_folio); int __lock_page_killable(struct page *__page) { @@ -1573,10 +1572,10 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm, return 0; } } else { - __lock_page(page); + __lock_folio(page_folio(page)); } - return 1; + return 1; } /** @@ -2720,7 +2719,9 @@ loff_t mapping_seek_hole_data(struct address_space *mapping, loff_t start, static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page, struct file **fpin) { - if (trylock_page(page)) + struct folio *folio = page_folio(page); + + if (trylock_folio(folio)) return 1; /* @@ -2733,7 +2734,7 @@ static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page, *fpin = maybe_unlock_mmap_for_io(vmf, *fpin); if (vmf->flags & FAULT_FLAG_KILLABLE) { - if (__lock_page_killable(page)) { + if (__lock_page_killable(&folio->page)) { /* * We didn't have the right flags to drop the mmap_lock, * but all fault_handlers only check for fatal signals @@ -2745,11 +2746,11 @@ static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page, return 0; } } else - __lock_page(page); + __lock_folio(folio); + return 1; } - /* * Synchronous readahead happens when we don't even find a page in the page * cache at all. We don't want to perform IO under the mmap sem, so if we have From patchwork Fri Mar 5 04:18:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A538C433E0 for ; Fri, 5 Mar 2021 04:23:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E16BA6500F for ; Fri, 5 Mar 2021 04:23:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E16BA6500F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7E2726B000D; Thu, 4 Mar 2021 23:23:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 792106B000E; Thu, 4 Mar 2021 23:23:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65AA86B0010; Thu, 4 Mar 2021 23:23:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0135.hostedemail.com [216.40.44.135]) by kanga.kvack.org (Postfix) with ESMTP id 49B166B000D for ; Thu, 4 Mar 2021 23:23:53 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 027DC8249980 for ; Fri, 5 Mar 2021 04:23:53 +0000 (UTC) X-FDA: 77884527546.03.A7B890E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id EB673E0011E2 for ; Fri, 5 Mar 2021 04:23:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=rUVZJhwmDMJXrM5/kmSBLRi/ckBRLZyv3cl88fxPOSA=; b=RIp167tW1vYjH7vCYEldVzSoHg VrN7UXt4NLMKRpVFypd+h3+Zl3YN+vs53AUX2nQj3DkiQdoSoa+hZyjKBslSLxiqxjDdo52iFBQhF GygrPvkSx4E/kFCShjBnni3aIifrD/136VRM1Htj/de4w6QS69IjF9V4Mkft5Q8rnJTTrqUALY+W7 /FzQI03qX+Z4rClNdJ7tM/CdYKtajvXaMU8D1GV+cgpEVwS5pb0e/M69Q+AxdqqoFlNyGOJuM+L+B JL2SGw60zxhE6E8f7/GToBJ1DKZz3sFGbLWGx/JlSMgy97iV4CR9IOxFqZk5k6UDw9xwETkXOID4W VCH9DAYQ==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1zh-00A3mt-Fl; Fri, 05 Mar 2021 04:23:26 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 14/25] mm/filemap: Add lock_folio_killable Date: Fri, 5 Mar 2021 04:18:50 +0000 Message-Id: <20210305041901.2396498-15-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: EB673E0011E2 X-Stat-Signature: 36ocibyr35z9nhq6q4hsr8gxhbq5ern8 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918229-680581 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is like lock_page_killable() but for use by callers who know they have a folio. Convert __lock_page_killable() to be __lock_folio_killable(). This saves one call to compound_head() per contended call to lock_page_killable(). __lock_folio_killable() is 20 bytes smaller than __lock_page_killable() was. lock_page_maybe_drop_mmap() shrinks by 68 bytes and __lock_page_or_retry() shrinks by 66 bytes. That's a total of 154 bytes of text saved. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 15 ++++++++++----- mm/filemap.c | 17 +++++++++-------- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 034e41256340..0fa1a0338e54 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -636,7 +636,7 @@ static inline bool wake_page_match(struct wait_page_queue *wait_page, } void __lock_folio(struct folio *folio); -extern int __lock_page_killable(struct page *page); +int __lock_folio_killable(struct folio *folio); extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags); @@ -676,6 +676,14 @@ static inline void lock_page(struct page *page) __lock_folio(folio); } +static inline int lock_folio_killable(struct folio *folio) +{ + might_sleep(); + if (!trylock_folio(folio)) + return __lock_folio_killable(folio); + return 0; +} + /* * lock_page_killable is like lock_page but can be interrupted by fatal * signals. It returns 0 if it locked the page and -EINTR if it was @@ -683,10 +691,7 @@ static inline void lock_page(struct page *page) */ static inline int lock_page_killable(struct page *page) { - might_sleep(); - if (!trylock_page(page)) - return __lock_page_killable(page); - return 0; + return lock_folio_killable(page_folio(page)); } /* diff --git a/mm/filemap.c b/mm/filemap.c index 3e3e3c666b94..5acadffed25d 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1499,14 +1499,13 @@ void __lock_folio(struct folio *folio) } EXPORT_SYMBOL(__lock_folio); -int __lock_page_killable(struct page *__page) +int __lock_folio_killable(struct folio *folio) { - struct page *page = compound_head(__page); - wait_queue_head_t *q = page_waitqueue(page); - return wait_on_page_bit_common(q, page, PG_locked, TASK_KILLABLE, + wait_queue_head_t *q = page_waitqueue(&folio->page); + return wait_on_page_bit_common(q, &folio->page, PG_locked, TASK_KILLABLE, EXCLUSIVE); } -EXPORT_SYMBOL_GPL(__lock_page_killable); +EXPORT_SYMBOL_GPL(__lock_folio_killable); int __lock_page_async(struct page *page, struct wait_page_queue *wait) { @@ -1548,6 +1547,8 @@ int __lock_page_async(struct page *page, struct wait_page_queue *wait) int __lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags) { + struct folio *folio = page_folio(page); + if (fault_flag_allow_retry_first(flags)) { /* * CAUTION! In this case, mmap_lock is not released @@ -1566,13 +1567,13 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm, if (flags & FAULT_FLAG_KILLABLE) { int ret; - ret = __lock_page_killable(page); + ret = __lock_folio_killable(folio); if (ret) { mmap_read_unlock(mm); return 0; } } else { - __lock_folio(page_folio(page)); + __lock_folio(folio); } return 1; @@ -2734,7 +2735,7 @@ static int lock_page_maybe_drop_mmap(struct vm_fault *vmf, struct page *page, *fpin = maybe_unlock_mmap_for_io(vmf, *fpin); if (vmf->flags & FAULT_FLAG_KILLABLE) { - if (__lock_page_killable(&folio->page)) { + if (__lock_folio_killable(folio)) { /* * We didn't have the right flags to drop the mmap_lock, * but all fault_handlers only check for fatal signals From patchwork Fri Mar 5 04:18:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC277C433DB for ; Fri, 5 Mar 2021 04:24:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7161764FE4 for ; Fri, 5 Mar 2021 04:24:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7161764FE4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 08E596B0008; Thu, 4 Mar 2021 23:24:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 03C6C6B000A; Thu, 4 Mar 2021 23:24:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1EF96B000E; Thu, 4 Mar 2021 23:24:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0157.hostedemail.com [216.40.44.157]) by kanga.kvack.org (Postfix) with ESMTP id C46676B0008 for ; Thu, 4 Mar 2021 23:24:38 -0500 (EST) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 869D2180ACC44 for ; Fri, 5 Mar 2021 04:24:38 +0000 (UTC) X-FDA: 77884529436.05.C68A00F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id DBA98A0000FA for ; Fri, 5 Mar 2021 04:24:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9d+mCHMlGcGG93YO8zcxGxoaCO1Ob3kebJ3Y/BX0vCc=; b=IIGNZaLDlocbNOcWiZR2oI2902 +OZ55+HOrai+yAVmwk2XoY0yL7yta2iyZ+Ght67SpbXgE0OOlM3MG4a8hKR44x7Aj97+TbU6WHlmv /RS66N1HAb7AeRvuNdDeE51EoM5X5wT9+FVl5dR503b5bZGc+r4a/qFKy9zranoD/wQckeVdiy72x aE4+2X/hyFqfyLcraQVDNYhP0Y/3btPMiP95E5/GDNnmWMjAvkCfrIT1eEXIPdk4LM9cReJ2PYedB Fh7ramj855Ib1FaLyAuLzqXJpvsBRoFHwXVfS7bVKMgNNk267WoOqfs311kLb+Q5o0hq5FCvNXFh0 CktHqZ/g==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI204-00A3oB-Sv; Fri, 05 Mar 2021 04:23:45 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 15/25] mm/filemap: Convert lock_page_async to lock_folio_async Date: Fri, 5 Mar 2021 04:18:51 +0000 Message-Id: <20210305041901.2396498-16-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: DBA98A0000FA X-Stat-Signature: 4gfw68o4znqrsn5f7j3qjn9pp8r1m3qj Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918277-314130 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There aren't any actual callers of lock_page_async(), but convert filemap_update_page() to call __lock_folio_async(). __lock_folio_async() is 21 bytes smaller than __lock_page_async(), but the real savings come from using a folio in filemap_update_page(), shrinking it from 514 bytes to 403 bytes, saving 111 bytes. The text shrinks by 132 bytes in total. Signed-off-by: Matthew Wilcox (Oracle) --- fs/io_uring.c | 2 +- include/linux/pagemap.h | 14 +++++++------- mm/filemap.c | 31 ++++++++++++++++--------------- 3 files changed, 24 insertions(+), 23 deletions(-) diff --git a/fs/io_uring.c b/fs/io_uring.c index 4a088581b0f2..55687707b5fb 100644 --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -3155,7 +3155,7 @@ static int io_read_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) } /* - * This is our waitqueue callback handler, registered through lock_page_async() + * This is our waitqueue callback handler, registered through lock_folio_async() * when we initially tried to do the IO with the iocb armed our waitqueue. * This gets called when the page is unlocked, and we generally expect that to * happen when the page IO is completed and the page is now uptodate. This will diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 0fa1a0338e54..9dbd9cf7d541 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -637,7 +637,7 @@ static inline bool wake_page_match(struct wait_page_queue *wait_page, void __lock_folio(struct folio *folio); int __lock_folio_killable(struct folio *folio); -extern int __lock_page_async(struct page *page, struct wait_page_queue *wait); +int __lock_folio_async(struct folio *folio, struct wait_page_queue *wait); extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags); void unlock_page(struct page *page); @@ -695,18 +695,18 @@ static inline int lock_page_killable(struct page *page) } /* - * lock_page_async - Lock the page, unless this would block. If the page - * is already locked, then queue a callback when the page becomes unlocked. + * lock_folio_async - Lock the folio, unless this would block. If the folio + * is already locked, then queue a callback when the folio becomes unlocked. * This callback can then retry the operation. * - * Returns 0 if the page is locked successfully, or -EIOCBQUEUED if the page + * Returns 0 if the folio is locked successfully, or -EIOCBQUEUED if the folio * was already locked and the callback defined in 'wait' was queued. */ -static inline int lock_page_async(struct page *page, +static inline int lock_folio_async(struct folio *folio, struct wait_page_queue *wait) { - if (!trylock_page(page)) - return __lock_page_async(page, wait); + if (!trylock_folio(folio)) + return __lock_folio_async(folio, wait); return 0; } diff --git a/mm/filemap.c b/mm/filemap.c index 5acadffed25d..b99b068bc058 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1507,18 +1507,18 @@ int __lock_folio_killable(struct folio *folio) } EXPORT_SYMBOL_GPL(__lock_folio_killable); -int __lock_page_async(struct page *page, struct wait_page_queue *wait) +int __lock_folio_async(struct folio *folio, struct wait_page_queue *wait) { - struct wait_queue_head *q = page_waitqueue(page); + struct wait_queue_head *q = page_waitqueue(&folio->page); int ret = 0; - wait->page = page; + wait->page = &folio->page; wait->bit_nr = PG_locked; spin_lock_irq(&q->lock); __add_wait_queue_entry_tail(q, &wait->wait); - SetPageWaiters(page); - ret = !trylock_page(page); + SetFolioWaiters(folio); + ret = !trylock_folio(folio); /* * If we were successful now, we know we're still on the * waitqueue as we're still under the lock. This means it's @@ -2265,41 +2265,42 @@ static int filemap_update_page(struct kiocb *iocb, struct address_space *mapping, struct iov_iter *iter, struct page *page) { + struct folio *folio = page_folio(page); int error; - if (!trylock_page(page)) { + if (!trylock_folio(folio)) { if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_NOIO)) return -EAGAIN; if (!(iocb->ki_flags & IOCB_WAITQ)) { - put_and_wait_on_page_locked(page, TASK_KILLABLE); + put_and_wait_on_page_locked(&folio->page, TASK_KILLABLE); return AOP_TRUNCATED_PAGE; } - error = __lock_page_async(page, iocb->ki_waitq); + error = __lock_folio_async(folio, iocb->ki_waitq); if (error) return error; } - if (!page->mapping) + if (!folio->page.mapping) goto truncated; error = 0; - if (filemap_range_uptodate(mapping, iocb->ki_pos, iter, page)) + if (filemap_range_uptodate(mapping, iocb->ki_pos, iter, &folio->page)) goto unlock; error = -EAGAIN; if (iocb->ki_flags & (IOCB_NOIO | IOCB_NOWAIT | IOCB_WAITQ)) goto unlock; - error = filemap_read_page(iocb->ki_filp, mapping, page); + error = filemap_read_page(iocb->ki_filp, mapping, &folio->page); if (error == AOP_TRUNCATED_PAGE) - put_page(page); + put_folio(folio); return error; truncated: - unlock_page(page); - put_page(page); + unlock_folio(folio); + put_folio(folio); return AOP_TRUNCATED_PAGE; unlock: - unlock_page(page); + unlock_folio(folio); return error; } From patchwork Fri Mar 5 04:18:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E3D1C433E0 for ; Fri, 5 Mar 2021 04:25:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B44BB65009 for ; Fri, 5 Mar 2021 04:25:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B44BB65009 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4F8386B000A; Thu, 4 Mar 2021 23:25:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A8706B000C; Thu, 4 Mar 2021 23:25:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 321CA6B000D; Thu, 4 Mar 2021 23:25:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0244.hostedemail.com [216.40.44.244]) by kanga.kvack.org (Postfix) with ESMTP id 12D0F6B000A for ; Thu, 4 Mar 2021 23:25:11 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CED7712CF for ; Fri, 5 Mar 2021 04:25:10 +0000 (UTC) X-FDA: 77884530780.06.916EA57 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id DC228D7 for ; Fri, 5 Mar 2021 04:25:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=nazHYrrFDV44XE1641cI3TndwK0v+eGEx7jvUCN/rZc=; b=RzztMA15GaL6tV6cZcBJyxZadt uvuISHcz7hstZ+alA7qY58w7WIGHXhxju1mSJfflOPp7zfp+HijGdL9MFcvrExEtg0YOs0UzlHs+1 B7uG4t0Tu6Q+nIAHjJixI+IZ5cLgkqTGoVAXn9BtnbFeO+crvQ7zjHCAGbqVDKrLIYeWCFeHwa1lc jyHoS9r54aU7d9HdLgESYk5oT6cSOJrw29Rd10UYkP0o2HTGHhrKV6V8WQqVhBBvJb9rnb3Rgd7JB mQteW2BL4ifEwjS+eY/8CVqaMAVt32DpTsBY+3t7nq7gnZMh4z4GqXjgsQsw9Kjnf1J2QRuYbVhRh 5/LmRQNg==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI20R-00A3pO-0h; Fri, 05 Mar 2021 04:24:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 16/25] mm/filemap: Convert end_page_writeback to end_folio_writeback Date: Fri, 5 Mar 2021 04:18:52 +0000 Message-Id: <20210305041901.2396498-17-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: DC228D7 X-Stat-Signature: fej9bzur416jifsr3k9r8mgsrids4ub5 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918308-682633 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a wrapper function for users that are not yet converted to folios. end_folio_writeback() is less than half the size of end_page_writeback() at just 105 bytes compared to 213 bytes, due to removing all the compound_head() calls. The 30 byte wrapper function makes this a net saving of 70 bytes. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 3 ++- mm/filemap.c | 30 +++++++++++++++--------------- mm/folio-compat.c | 6 ++++++ 3 files changed, 23 insertions(+), 16 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 9dbd9cf7d541..e7fe8b276a2d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -753,7 +753,8 @@ static inline int wait_on_page_locked_killable(struct page *page) int put_and_wait_on_page_locked(struct page *page, int state); void wait_on_page_writeback(struct page *page); -extern void end_page_writeback(struct page *page); +void end_page_writeback(struct page *page); +void end_folio_writeback(struct folio *folio); void wait_for_stable_page(struct page *page); void page_endio(struct page *page, bool is_write, int err); diff --git a/mm/filemap.c b/mm/filemap.c index b99b068bc058..e25a5ebc914c 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1148,11 +1148,11 @@ static void wake_up_page_bit(struct page *page, int bit_nr) spin_unlock_irqrestore(&q->lock, flags); } -static void wake_up_page(struct page *page, int bit) +static void wake_up_folio(struct folio *folio, int bit) { - if (!PageWaiters(page)) + if (!FolioWaiters(folio)) return; - wake_up_page_bit(page, bit); + wake_up_page_bit(&folio->page, bit); } /* @@ -1426,10 +1426,10 @@ void unlock_folio(struct folio *folio) EXPORT_SYMBOL(unlock_folio); /** - * end_page_writeback - end writeback against a page - * @page: the page + * end_folio_writeback - End writeback against a page. + * @folio: The page. */ -void end_page_writeback(struct page *page) +void end_folio_writeback(struct folio *folio) { /* * TestClearPageReclaim could be used here but it is an atomic @@ -1438,26 +1438,26 @@ void end_page_writeback(struct page *page) * justify taking an atomic operation penalty at the end of * ever page writeback. */ - if (PageReclaim(page)) { - ClearPageReclaim(page); - rotate_reclaimable_page(page); + if (FolioReclaim(folio)) { + ClearFolioReclaim(folio); + rotate_reclaimable_page(&folio->page); } /* * Writeback does not hold a page reference of its own, relying * on truncation to wait for the clearing of PG_writeback. * But here we must make sure that the page is not freed and - * reused before the wake_up_page(). + * reused before the wake_up_folio(). */ - get_page(page); - if (!test_clear_page_writeback(page)) + get_folio(folio); + if (!test_clear_page_writeback(&folio->page)) BUG(); smp_mb__after_atomic(); - wake_up_page(page, PG_writeback); - put_page(page); + wake_up_folio(folio, PG_writeback); + put_folio(folio); } -EXPORT_SYMBOL(end_page_writeback); +EXPORT_SYMBOL(end_folio_writeback); /* * After completing I/O on a page, call this routine to update the page diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 02798abf19a1..d1a1dfe52589 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -17,3 +17,9 @@ void unlock_page(struct page *page) return unlock_folio(page_folio(page)); } EXPORT_SYMBOL(unlock_page); + +void end_page_writeback(struct page *page) +{ + return end_folio_writeback(page_folio(page)); +} +EXPORT_SYMBOL(end_page_writeback); From patchwork Fri Mar 5 04:18:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC222C433DB for ; Fri, 5 Mar 2021 04:25:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9B71765009 for ; Fri, 5 Mar 2021 04:25:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9B71765009 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9DB1A6B000C; Thu, 4 Mar 2021 23:25:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9624F6B000E; Thu, 4 Mar 2021 23:25:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 76B196B000C; Thu, 4 Mar 2021 23:25:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0008.hostedemail.com [216.40.44.8]) by kanga.kvack.org (Postfix) with ESMTP id 531926B000D for ; Thu, 4 Mar 2021 23:25:11 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 168BF8249980 for ; Fri, 5 Mar 2021 04:25:11 +0000 (UTC) X-FDA: 77884530822.11.760BE87 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf05.hostedemail.com (Postfix) with ESMTP id 111E0E0011E0 for ; Fri, 5 Mar 2021 04:25:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=TbwKTsRFeRRDANTFRahEyrvCD2paHgaZGJcv04iRdsE=; b=pPmEabNdKjB2yes6KuAwcqe3iJ bh5J/cXoSJsZvy/YKVKrwJRFMYWcepswsJW28fGrnwftIMY8F9JC/oCMFUAA55u4S4ViSVRFfBPP5 yCKnvtXvECap+Wj2ue5CnVjDyYGMC+eANie/Ih7Qw/xTplhBSydoVTa1gCngKVlWx9h+Zeco4FA3Q bGTRWF38KztlWT1KaQKxr9+lXMCM2UYnd4vs5a7fiSPWmYA9Vl0SIt03wdBDbU80MrIyOJskBDmVR UP/KrOv2zin6BARrEDTAnkZMW8Slq1YebYC9IOX11T/LrdjM18ZAbvc7yu9NfJQtjLEyo33U/AG1l GlF0JACA==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI20v-00A3tR-4l; Fri, 05 Mar 2021 04:24:38 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 17/25] mm/filemap: Add wait_on_folio_locked & wait_on_folio_locked_killable Date: Fri, 5 Mar 2021 04:18:53 +0000 Message-Id: <20210305041901.2396498-18-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 111E0E0011E0 X-Stat-Signature: s473hfx8pc1d9hubaztrd9ugwm8pseni Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf05; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918309-212155 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Turn wait_on_page_locked() and wait_on_page_locked_killable() into wrappers. This eliminates a call to compound_head() from each call-site, reducing text size by 207 bytes for me. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 26 ++++++++++++++++++-------- mm/filemap.c | 4 ++-- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index e7fe8b276a2d..c53743f24550 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -732,23 +732,33 @@ extern void wait_on_page_bit(struct page *page, int bit_nr); extern int wait_on_page_bit_killable(struct page *page, int bit_nr); /* - * Wait for a page to be unlocked. + * Wait for a folio to be unlocked. * - * This must be called with the caller "holding" the page, - * ie with increased "page->count" so that the page won't + * This must be called with the caller "holding" the folio, + * ie with increased "page->count" so that the folio won't * go away during the wait.. */ +static inline void wait_on_folio_locked(struct folio *folio) +{ + if (FolioLocked(folio)) + wait_on_page_bit(&folio->page, PG_locked); +} + +static inline int wait_on_folio_locked_killable(struct folio *folio) +{ + if (!FolioLocked(folio)) + return 0; + return wait_on_page_bit_killable(&folio->page, PG_locked); +} + static inline void wait_on_page_locked(struct page *page) { - if (PageLocked(page)) - wait_on_page_bit(compound_head(page), PG_locked); + wait_on_folio_locked(page_folio(page)); } static inline int wait_on_page_locked_killable(struct page *page) { - if (!PageLocked(page)) - return 0; - return wait_on_page_bit_killable(compound_head(page), PG_locked); + return wait_on_folio_locked_killable(page_folio(page)); } int put_and_wait_on_page_locked(struct page *page, int state); diff --git a/mm/filemap.c b/mm/filemap.c index e25a5ebc914c..50263fa62574 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1559,9 +1559,9 @@ int __lock_page_or_retry(struct page *page, struct mm_struct *mm, mmap_read_unlock(mm); if (flags & FAULT_FLAG_KILLABLE) - wait_on_page_locked_killable(page); + wait_on_folio_locked_killable(folio); else - wait_on_page_locked(page); + wait_on_folio_locked(folio); return 0; } if (flags & FAULT_FLAG_KILLABLE) { From patchwork Fri Mar 5 04:18:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4081FC433E6 for ; Fri, 5 Mar 2021 04:25:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DA99865009 for ; Fri, 5 Mar 2021 04:25:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DA99865009 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F2FB6B000D; Thu, 4 Mar 2021 23:25:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 67BCB6B000E; Thu, 4 Mar 2021 23:25:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F68C6B0010; Thu, 4 Mar 2021 23:25:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0080.hostedemail.com [216.40.44.80]) by kanga.kvack.org (Postfix) with ESMTP id 2F93E6B000D for ; Thu, 4 Mar 2021 23:25:17 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E9E25181B04B3 for ; Fri, 5 Mar 2021 04:25:16 +0000 (UTC) X-FDA: 77884531032.18.72C4091 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf01.hostedemail.com (Postfix) with ESMTP id ACB772000382 for ; Fri, 5 Mar 2021 04:25:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ByBvZRAgYRACtY2rVghjrF+s5V0nWEFj/GUpqmY/krg=; b=bdhen0e0A+TMa408MU8/lgkXfj tjTiCKz2w10Kbk5JviHWXpBE+ZxYrTDcBZDzM5esN+e1I2n44YqOa4Le8lUl/Ky/CoVNpWQ1XQbty 65cLZ8HbD/GcFhxKjmU/W5DBmdMhriRhVi3WkLBW4I8MuCCZGVhP/bqE3O5Y7vDCvByMa0IPN2vqS cOjr3BpbSvWV1LWwYZKg1Ufd6BBgPn86nGGLlrGhYNeo5sLiM4cI4W2LyoS39JSchQYsdoA2pK8fm KP/HWKLkt93OIea0kGGllnKQccsmjdLu5JYm0NsRnYTJDW0DeLM7OWcu6UzFg8TD9+Jj/chiHvV++ FcIiqP/A==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI216-00A409-R4; Fri, 05 Mar 2021 04:24:48 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 18/25] mm/page-writeback: Add wait_on_folio_writeback Date: Fri, 5 Mar 2021 04:18:54 +0000 Message-Id: <20210305041901.2396498-19-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: azdeuakrmnjbjq9c1othnhhtfwuwo3eo X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: ACB772000382 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf01; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918316-448672 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This eliminates a call to compound_head() by turning PageWriteback() into FolioWriteback(), which saves 8 bytes. That is more than offset by adding the wait_on_page_writeback compatibility wrapper for a net increase in text of 30 bytes. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 1 + mm/folio-compat.c | 6 ++++++ mm/page-writeback.c | 21 ++++++++++++++------- 3 files changed, 21 insertions(+), 7 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index c53743f24550..7d797847633c 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -763,6 +763,7 @@ static inline int wait_on_page_locked_killable(struct page *page) int put_and_wait_on_page_locked(struct page *page, int state); void wait_on_page_writeback(struct page *page); +void wait_on_folio_writeback(struct folio *folio); void end_page_writeback(struct page *page); void end_folio_writeback(struct folio *folio); void wait_for_stable_page(struct page *page); diff --git a/mm/folio-compat.c b/mm/folio-compat.c index d1a1dfe52589..6aadecc39fba 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -23,3 +23,9 @@ void end_page_writeback(struct page *page) return end_folio_writeback(page_folio(page)); } EXPORT_SYMBOL(end_page_writeback); + +void wait_on_page_writeback(struct page *page) +{ + return wait_on_folio_writeback(page_folio(page)); +} +EXPORT_SYMBOL_GPL(wait_on_page_writeback); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index eb34d204d4ee..968579452ea4 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2821,17 +2821,24 @@ int __test_set_page_writeback(struct page *page, bool keep_write) } EXPORT_SYMBOL(__test_set_page_writeback); -/* - * Wait for a page to complete writeback +/** + * wait_on_folio_writeback - Wait for a folio to complete writeback. + * @folio: The folio to wait for. + * + * If the folio is currently being written back to storage, waits for the + * I/O to complete. + * + * Context: Sleeps; must be called in process context and with no spinlocks + * held. */ -void wait_on_page_writeback(struct page *page) +void wait_on_folio_writeback(struct folio *folio) { - while (PageWriteback(page)) { - trace_wait_on_page_writeback(page, page_mapping(page)); - wait_on_page_bit(page, PG_writeback); + while (FolioWriteback(folio)) { + trace_wait_on_page_writeback(&folio->page, folio_mapping(folio)); + wait_on_page_bit(&folio->page, PG_writeback); } } -EXPORT_SYMBOL_GPL(wait_on_page_writeback); +EXPORT_SYMBOL_GPL(wait_on_folio_writeback); /** * wait_for_stable_page() - wait for writeback to finish, if necessary. From patchwork Fri Mar 5 04:18:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6A5BC433DB for ; Fri, 5 Mar 2021 04:25:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5D18664FE4 for ; Fri, 5 Mar 2021 04:25:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D18664FE4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E45A26B0008; Thu, 4 Mar 2021 23:25:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DCE106B000E; Thu, 4 Mar 2021 23:25:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C48966B0010; Thu, 4 Mar 2021 23:25:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id A44736B0008 for ; Thu, 4 Mar 2021 23:25:42 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 66B2F8249980 for ; Fri, 5 Mar 2021 04:25:42 +0000 (UTC) X-FDA: 77884532124.23.4C8D0A7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id 5D5572000381 for ; Fri, 5 Mar 2021 04:25:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=ViUiDGdVnd7Iq4vdJDnK9pV32OXyBauhh7HNXcs3hWk=; b=HJV/2O4pf9RXMAw6lAoKTDf54y VEGFEKf0LEZCoauqkXpByXvcTOBxs0+GGYG3YARlLfXTahsDu1B/nhhDu0qkye3eO+/ygcRYy5K5X yv4f+ePzXS2tofWUz1fM6yCkOH4jouoL6hf2d/iHfjBTnKKywDXsNgWzqyOkT6u0zWThHeZubs51u JumpTVB5BnzgeHQ/MPt9YM+MbWoN5OU5DrLqfusBrUp6gqJpD/HMd4ahE/spyuNs8aBYKfpKsqc5v O+osyB0Cqf0VgOfm9QRnmmQ21abHsT/jwZqSUUYK19imKb0akySFhqSId9oaXLM+fFolZAGbRoQHF ui0+xE8g==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI21V-00A48z-E9; Fri, 05 Mar 2021 04:25:12 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 19/25] mm/page-writeback: Add wait_for_stable_folio Date: Fri, 5 Mar 2021 04:18:55 +0000 Message-Id: <20210305041901.2396498-20-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 5D5572000381 X-Stat-Signature: 11fssnt95cgque6srrg9anw1aopcsgmm Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf28; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918342-555069 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move wait_for_stable_page() into the folio compatibility file. wait_for_stable_folio() avoids a call to compound_head() and is 14 bytes smaller than wait_for_stable_page() was. The net text size grows by 24 bytes as a result of this patch. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 1 + mm/folio-compat.c | 6 ++++++ mm/page-writeback.c | 17 ++++++++--------- 3 files changed, 15 insertions(+), 9 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 7d797847633c..e7ca8def2f0d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -767,6 +767,7 @@ void wait_on_folio_writeback(struct folio *folio); void end_page_writeback(struct page *page); void end_folio_writeback(struct folio *folio); void wait_for_stable_page(struct page *page); +void wait_for_stable_folio(struct folio *folio); void page_endio(struct page *page, bool is_write, int err); diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 6aadecc39fba..335594fe414e 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -29,3 +29,9 @@ void wait_on_page_writeback(struct page *page) return wait_on_folio_writeback(page_folio(page)); } EXPORT_SYMBOL_GPL(wait_on_page_writeback); + +void wait_for_stable_page(struct page *page) +{ + return wait_for_stable_folio(page_folio(page)); +} +EXPORT_SYMBOL_GPL(wait_for_stable_page); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 968579452ea4..eef36edc9e0c 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2841,17 +2841,16 @@ void wait_on_folio_writeback(struct folio *folio) EXPORT_SYMBOL_GPL(wait_on_folio_writeback); /** - * wait_for_stable_page() - wait for writeback to finish, if necessary. - * @page: The page to wait on. + * wait_for_stable_folio() - wait for writeback to finish, if necessary. + * @folio: The folio to wait on. * - * This function determines if the given page is related to a backing device - * that requires page contents to be held stable during writeback. If so, then + * This function determines if the given folio is related to a backing device + * that requires folio contents to be held stable during writeback. If so, then * it will wait for any pending writeback to complete. */ -void wait_for_stable_page(struct page *page) +void wait_for_stable_folio(struct folio *folio) { - page = thp_head(page); - if (page->mapping->host->i_sb->s_iflags & SB_I_STABLE_WRITES) - wait_on_page_writeback(page); + if (folio->page.mapping->host->i_sb->s_iflags & SB_I_STABLE_WRITES) + wait_on_folio_writeback(folio); } -EXPORT_SYMBOL_GPL(wait_for_stable_page); +EXPORT_SYMBOL_GPL(wait_for_stable_folio); From patchwork Fri Mar 5 04:18:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADA39C433E0 for ; Fri, 5 Mar 2021 04:26:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3E35E65009 for ; Fri, 5 Mar 2021 04:26:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3E35E65009 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C979C6B000A; Thu, 4 Mar 2021 23:26:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C6E066B000E; Thu, 4 Mar 2021 23:26:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE7896B0012; Thu, 4 Mar 2021 23:26:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0088.hostedemail.com [216.40.44.88]) by kanga.kvack.org (Postfix) with ESMTP id 8F9C86B000E for ; Thu, 4 Mar 2021 23:26:13 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4BDF43658 for ; Fri, 5 Mar 2021 04:26:13 +0000 (UTC) X-FDA: 77884533426.19.F5DD69C Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf30.hostedemail.com (Postfix) with ESMTP id 27F02E0011DD for ; Fri, 5 Mar 2021 04:26:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=A240a8RFlOLCpB1wjQo7mwwJUGZkhVYLRXzrJQR2XmA=; b=g0KWz96LnDBzndIXXzd8dvEQ56 LEH5ZFAbtdATCWQxf4jQ45SE20bfh8mEKCeJLp9Oot0HQtKbfqdUQmGaQf1+Jlzz8uWNNRCQPY30E EnQdBK01XKOP113YHg0qEIwgKPYEsNPLinQkEBFr/zXd2wb347nKpn9IOR3ahIxtR3zDapG4QYPlk QEuCM/RGY69Ox8EDiQJx+z3Rk7Up3BP+8Flt1SQr7vf1yJCE9ITNv0MYi0B1MJqEcFleJjgqMejRJ AnzIEqt1xjbHHMPPwzQ/7eZ6jx3M4aAKFGDU4jKtlevQf2+bEOq8/yIg9OnB6STmDwXce3AXufgnx Dm4vrbxg==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI21d-00A4ff-94; Fri, 05 Mar 2021 04:25:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 20/25] mm/filemap: Convert wait_on_page_bit to wait_on_folio_bit Date: Fri, 5 Mar 2021 04:18:56 +0000 Message-Id: <20210305041901.2396498-21-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 27F02E0011DD X-Stat-Signature: uoygwmdtxsjyb58qanygntxjsrn1ozae Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf30; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918370-442861 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We must always wait on the folio, otherwise we won't be woken up. This commit shrinks the kernel by 695 bytes, mostly due to moving the page waitqueue lookup into wait_on_folio_bit_common(). Signed-off-by: Matthew Wilcox (Oracle) --- fs/afs/write.c | 21 ++++++++-------- include/linux/pagemap.h | 10 ++++---- mm/filemap.c | 56 ++++++++++++++++++----------------------- mm/page-writeback.c | 2 +- 4 files changed, 42 insertions(+), 47 deletions(-) diff --git a/fs/afs/write.c b/fs/afs/write.c index c9195fc67fd8..0c313117200a 100644 --- a/fs/afs/write.c +++ b/fs/afs/write.c @@ -834,13 +834,14 @@ int afs_fsync(struct file *file, loff_t start, loff_t end, int datasync) */ vm_fault_t afs_page_mkwrite(struct vm_fault *vmf) { + struct folio *folio = page_folio(vmf->page); struct file *file = vmf->vma->vm_file; struct inode *inode = file_inode(file); struct afs_vnode *vnode = AFS_FS_I(inode); unsigned long priv; _enter("{{%llx:%llu}},{%lx}", - vnode->fid.vid, vnode->fid.vnode, vmf->page->index); + vnode->fid.vid, vnode->fid.vnode, folio->page.index); sb_start_pagefault(inode->i_sb); @@ -851,27 +852,27 @@ vm_fault_t afs_page_mkwrite(struct vm_fault *vmf) fscache_wait_on_page_write(vnode->cache, vmf->page); #endif - if (PageWriteback(vmf->page) && - wait_on_page_bit_killable(vmf->page, PG_writeback) < 0) + if (FolioWriteback(folio) && + wait_on_folio_bit_killable(folio, PG_writeback) < 0) return VM_FAULT_RETRY; - if (lock_page_killable(vmf->page) < 0) + if (lock_folio_killable(folio) < 0) return VM_FAULT_RETRY; /* We mustn't change page->private until writeback is complete as that * details the portion of the page we need to write back and we might * need to redirty the page if there's a problem. */ - wait_on_page_writeback(vmf->page); + wait_on_folio_writeback(folio); - priv = afs_page_dirty(0, PAGE_SIZE); + priv = afs_page_dirty(0, folio_size(folio)); priv = afs_page_dirty_mmapped(priv); trace_afs_page_dirty(vnode, tracepoint_string("mkwrite"), - vmf->page->index, priv); - if (PagePrivate(vmf->page)) - set_page_private(vmf->page, priv); + folio->page.index, priv); + if (FolioPrivate(folio)) + set_folio_private(folio, priv); else - attach_page_private(vmf->page, (void *)priv); + attach_folio_private(folio, (void *)priv); file_update_time(file); sb_end_pagefault(inode->i_sb); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index e7ca8def2f0d..8737eb86602e 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -725,11 +725,11 @@ static inline int lock_page_or_retry(struct page *page, struct mm_struct *mm, } /* - * This is exported only for wait_on_page_locked/wait_on_page_writeback, etc., + * This is exported only for wait_on_folio_locked/wait_on_folio_writeback, etc., * and should not be used directly. */ -extern void wait_on_page_bit(struct page *page, int bit_nr); -extern int wait_on_page_bit_killable(struct page *page, int bit_nr); +extern void wait_on_folio_bit(struct folio *folio, int bit_nr); +extern int wait_on_folio_bit_killable(struct folio *folio, int bit_nr); /* * Wait for a folio to be unlocked. @@ -741,14 +741,14 @@ extern int wait_on_page_bit_killable(struct page *page, int bit_nr); static inline void wait_on_folio_locked(struct folio *folio) { if (FolioLocked(folio)) - wait_on_page_bit(&folio->page, PG_locked); + wait_on_folio_bit(folio, PG_locked); } static inline int wait_on_folio_locked_killable(struct folio *folio) { if (!FolioLocked(folio)) return 0; - return wait_on_page_bit_killable(&folio->page, PG_locked); + return wait_on_folio_bit_killable(folio, PG_locked); } static inline void wait_on_page_locked(struct page *page) diff --git a/mm/filemap.c b/mm/filemap.c index 50263fa62574..ec5a743b4e0f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1075,7 +1075,7 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, * * So update the flags atomically, and wake up the waiter * afterwards to avoid any races. This store-release pairs - * with the load-acquire in wait_on_page_bit_common(). + * with the load-acquire in wait_on_folio_bit_common(). */ smp_store_release(&wait->flags, flags | WQ_FLAG_WOKEN); wake_up_state(wait->private, mode); @@ -1156,7 +1156,7 @@ static void wake_up_folio(struct folio *folio, int bit) } /* - * A choice of three behaviors for wait_on_page_bit_common(): + * A choice of three behaviors for wait_on_folio_bit_common(): */ enum behavior { EXCLUSIVE, /* Hold ref to page and take the bit when woken, like @@ -1190,9 +1190,10 @@ static inline bool trylock_page_bit_common(struct page *page, int bit_nr, /* How many times do we accept lock stealing from under a waiter? */ int sysctl_page_lock_unfairness = 5; -static inline int wait_on_page_bit_common(wait_queue_head_t *q, - struct page *page, int bit_nr, int state, enum behavior behavior) +static inline int wait_on_folio_bit_common(struct folio *folio, int bit_nr, + int state, enum behavior behavior) { + wait_queue_head_t *q = page_waitqueue(&folio->page); int unfairness = sysctl_page_lock_unfairness; struct wait_page_queue wait_page; wait_queue_entry_t *wait = &wait_page.wait; @@ -1201,8 +1202,8 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, unsigned long pflags; if (bit_nr == PG_locked && - !PageUptodate(page) && PageWorkingset(page)) { - if (!PageSwapBacked(page)) { + !FolioUptodate(folio) && FolioWorkingset(folio)) { + if (!FolioSwapBacked(folio)) { delayacct_thrashing_start(); delayacct = true; } @@ -1212,7 +1213,7 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, init_wait(wait); wait->func = wake_page_function; - wait_page.page = page; + wait_page.page = &folio->page; wait_page.bit_nr = bit_nr; repeat: @@ -1227,7 +1228,7 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, * Do one last check whether we can get the * page bit synchronously. * - * Do the SetPageWaiters() marking before that + * Do the SetFolioWaiters() marking before that * to let any waker we _just_ missed know they * need to wake us up (otherwise they'll never * even go to the slow case that looks at the @@ -1238,8 +1239,8 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, * lock to avoid races. */ spin_lock_irq(&q->lock); - SetPageWaiters(page); - if (!trylock_page_bit_common(page, bit_nr, wait)) + SetFolioWaiters(folio); + if (!trylock_page_bit_common(&folio->page, bit_nr, wait)) __add_wait_queue_entry_tail(q, wait); spin_unlock_irq(&q->lock); @@ -1249,10 +1250,10 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, * see whether the page bit testing has already * been done by the wake function. * - * We can drop our reference to the page. + * We can drop our reference to the folio. */ if (behavior == DROP) - put_page(page); + put_folio(folio); /* * Note that until the "finish_wait()", or until @@ -1289,7 +1290,7 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, * * And if that fails, we'll have to retry this all. */ - if (unlikely(test_and_set_bit(bit_nr, &page->flags))) + if (unlikely(test_and_set_bit(bit_nr, folio_flags(folio)))) goto repeat; wait->flags |= WQ_FLAG_DONE; @@ -1298,7 +1299,7 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, /* * If a signal happened, this 'finish_wait()' may remove the last - * waiter from the wait-queues, but the PageWaiters bit will remain + * waiter from the wait-queues, but the FolioWaiters bit will remain * set. That's ok. The next wakeup will take care of it, and trying * to do it here would be difficult and prone to races. */ @@ -1329,19 +1330,17 @@ static inline int wait_on_page_bit_common(wait_queue_head_t *q, return wait->flags & WQ_FLAG_WOKEN ? 0 : -EINTR; } -void wait_on_page_bit(struct page *page, int bit_nr) +void wait_on_folio_bit(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(page); - wait_on_page_bit_common(q, page, bit_nr, TASK_UNINTERRUPTIBLE, SHARED); + wait_on_folio_bit_common(folio, bit_nr, TASK_UNINTERRUPTIBLE, SHARED); } -EXPORT_SYMBOL(wait_on_page_bit); +EXPORT_SYMBOL(wait_on_folio_bit); -int wait_on_page_bit_killable(struct page *page, int bit_nr) +int wait_on_folio_bit_killable(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(page); - return wait_on_page_bit_common(q, page, bit_nr, TASK_KILLABLE, SHARED); + return wait_on_folio_bit_common(folio, bit_nr, TASK_KILLABLE, SHARED); } -EXPORT_SYMBOL(wait_on_page_bit_killable); +EXPORT_SYMBOL(wait_on_folio_bit_killable); /** * put_and_wait_on_page_locked - Drop a reference and wait for it to be unlocked @@ -1358,11 +1357,8 @@ EXPORT_SYMBOL(wait_on_page_bit_killable); */ int put_and_wait_on_page_locked(struct page *page, int state) { - wait_queue_head_t *q; - - page = compound_head(page); - q = page_waitqueue(page); - return wait_on_page_bit_common(q, page, PG_locked, state, DROP); + return wait_on_folio_bit_common(page_folio(page), PG_locked, state, + DROP); } /** @@ -1493,16 +1489,14 @@ EXPORT_SYMBOL_GPL(page_endio); */ void __lock_folio(struct folio *folio) { - wait_queue_head_t *q = page_waitqueue(&folio->page); - wait_on_page_bit_common(q, &folio->page, PG_locked, TASK_UNINTERRUPTIBLE, + wait_on_folio_bit_common(folio, PG_locked, TASK_UNINTERRUPTIBLE, EXCLUSIVE); } EXPORT_SYMBOL(__lock_folio); int __lock_folio_killable(struct folio *folio) { - wait_queue_head_t *q = page_waitqueue(&folio->page); - return wait_on_page_bit_common(q, &folio->page, PG_locked, TASK_KILLABLE, + return wait_on_folio_bit_common(folio, PG_locked, TASK_KILLABLE, EXCLUSIVE); } EXPORT_SYMBOL_GPL(__lock_folio_killable); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index eef36edc9e0c..6c1b4737c383 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2835,7 +2835,7 @@ void wait_on_folio_writeback(struct folio *folio) { while (FolioWriteback(folio)) { trace_wait_on_page_writeback(&folio->page, folio_mapping(folio)); - wait_on_page_bit(&folio->page, PG_writeback); + wait_on_folio_bit(folio, PG_writeback); } } EXPORT_SYMBOL_GPL(wait_on_folio_writeback); From patchwork Fri Mar 5 04:18:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26660C433E6 for ; Fri, 5 Mar 2021 04:26:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C4C8F6500C for ; Fri, 5 Mar 2021 04:26:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C4C8F6500C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 40A206B0008; Thu, 4 Mar 2021 23:26:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B9B36B000C; Thu, 4 Mar 2021 23:26:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 25BFD6B000D; Thu, 4 Mar 2021 23:26:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0166.hostedemail.com [216.40.44.166]) by kanga.kvack.org (Postfix) with ESMTP id 09F4D6B0008 for ; Thu, 4 Mar 2021 23:26:52 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id C098D4403 for ; Fri, 5 Mar 2021 04:26:51 +0000 (UTC) X-FDA: 77884535022.06.19215ED Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id 5EB88200038F for ; Fri, 5 Mar 2021 04:26:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=CMOjg81XZerkssgDq+SP0LVRKbTXWxJnSrFjpOwOOZM=; b=Icu7EZK28GOfZsNWl87jyTE4j2 J5iOAHkQ/FjMiCLzY6C5/7U3MiqgcHYyd16QCFxZXbBYvtZVRRmA9jcpRHBaosC+Zfyf1ESpr0Ffo P1R8kdCK9kTyFWQ6cCpppT0VGzM8ArSEqqvp4EUdMJhvA5OvsHSfjjy4DFklHvro34FLEhWSiFL77 YR6aDunljvzbiwdVHPa+DqX9ZHx4jUmyCzE69NGQngs62HnOW1hZ0AjrFRgmhdCO6MomeK8+nPUsV cfuqOoxdW/Wdfx0bP4+x8S51TG0CL9HFiNcivAatJnRVs/UixlX9x7MzjLhcsl9dXQ1qpE/okl0ZK KJHU+qFA==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI22B-00A4hV-C6; Fri, 05 Mar 2021 04:26:04 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 21/25] mm/filemap: Add __lock_folio_or_retry Date: Fri, 5 Mar 2021 04:18:57 +0000 Message-Id: <20210305041901.2396498-22-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 5EB88200038F X-Stat-Signature: nuhzwhkrscheexfymwknukica4uzmo8n Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf11; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918409-211171 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert __lock_page_or_retry() to __lock_folio_or_retry(). This actually saves 4 bytes in the only caller of lock_page_or_retry() (due to better register allocation) and saves the 20 byte cost of calling page_folio() in __lock_folio_or_retry() for a total saving of 24 bytes. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 9 ++++++--- mm/filemap.c | 10 ++++------ mm/memory.c | 8 ++++---- 3 files changed, 14 insertions(+), 13 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 8737eb86602e..6ee4bc843f98 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -638,7 +638,7 @@ static inline bool wake_page_match(struct wait_page_queue *wait_page, void __lock_folio(struct folio *folio); int __lock_folio_killable(struct folio *folio); int __lock_folio_async(struct folio *folio, struct wait_page_queue *wait); -extern int __lock_page_or_retry(struct page *page, struct mm_struct *mm, +int __lock_folio_or_retry(struct folio *folio, struct mm_struct *mm, unsigned int flags); void unlock_page(struct page *page); void unlock_folio(struct folio *folio); @@ -715,13 +715,16 @@ static inline int lock_folio_async(struct folio *folio, * caller indicated that it can handle a retry. * * Return value and mmap_lock implications depend on flags; see - * __lock_page_or_retry(). + * __lock_folio_or_retry(). */ static inline int lock_page_or_retry(struct page *page, struct mm_struct *mm, unsigned int flags) { + struct folio *folio; might_sleep(); - return trylock_page(page) || __lock_page_or_retry(page, mm, flags); + + folio = page_folio(page); + return trylock_folio(folio) || __lock_folio_or_retry(folio, mm, flags); } /* diff --git a/mm/filemap.c b/mm/filemap.c index ec5a743b4e0f..76c97cb9cbbe 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1529,20 +1529,18 @@ int __lock_folio_async(struct folio *folio, struct wait_page_queue *wait) /* * Return values: - * 1 - page is locked; mmap_lock is still held. - * 0 - page is not locked. + * 1 - folio is locked; mmap_lock is still held. + * 0 - folio is not locked. * mmap_lock has been released (mmap_read_unlock(), unless flags had both * FAULT_FLAG_ALLOW_RETRY and FAULT_FLAG_RETRY_NOWAIT set, in * which case mmap_lock is still held. * * If neither ALLOW_RETRY nor KILLABLE are set, will always return 1 - * with the page locked and the mmap_lock unperturbed. + * with the folio locked and the mmap_lock unperturbed. */ -int __lock_page_or_retry(struct page *page, struct mm_struct *mm, +int __lock_folio_or_retry(struct folio *folio, struct mm_struct *mm, unsigned int flags) { - struct folio *folio = page_folio(page); - if (fault_flag_allow_retry_first(flags)) { /* * CAUTION! In this case, mmap_lock is not released diff --git a/mm/memory.c b/mm/memory.c index c8e357627318..51a3590418f0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4022,7 +4022,7 @@ static vm_fault_t do_shared_fault(struct vm_fault *vmf) * We enter with non-exclusive mmap_lock (to exclude vma changes, * but allow concurrent faults). * The mmap_lock may have been released depending on flags and our - * return value. See filemap_fault() and __lock_page_or_retry(). + * return value. See filemap_fault() and __lock_folio_or_retry(). * If mmap_lock is released, vma may become invalid (for example * by other thread calling munmap()). */ @@ -4256,7 +4256,7 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud) * concurrent faults). * * The mmap_lock may have been released depending on flags and our return value. - * See filemap_fault() and __lock_page_or_retry(). + * See filemap_fault() and __lock_folio_or_retry(). */ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) { @@ -4360,7 +4360,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * By the time we get here, we already hold the mm semaphore * * The mmap_lock may have been released depending on flags and our - * return value. See filemap_fault() and __lock_page_or_retry(). + * return value. See filemap_fault() and __lock_folio_or_retry(). */ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags) @@ -4516,7 +4516,7 @@ static inline void mm_account_fault(struct pt_regs *regs, * By the time we get here, we already hold the mm semaphore * * The mmap_lock may have been released depending on flags and our - * return value. See filemap_fault() and __lock_page_or_retry(). + * return value. See filemap_fault() and __lock_folio_or_retry(). */ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, unsigned int flags, struct pt_regs *regs) From patchwork Fri Mar 5 04:18:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66EDDC433DB for ; Fri, 5 Mar 2021 04:26:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 114A164FE4 for ; Fri, 5 Mar 2021 04:26:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 114A164FE4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A25D16B000C; Thu, 4 Mar 2021 23:26:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9FC1C6B000D; Thu, 4 Mar 2021 23:26:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C48A6B000E; Thu, 4 Mar 2021 23:26:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0168.hostedemail.com [216.40.44.168]) by kanga.kvack.org (Postfix) with ESMTP id 6C8316B000C for ; Thu, 4 Mar 2021 23:26:56 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 32FA78249980 for ; Fri, 5 Mar 2021 04:26:56 +0000 (UTC) X-FDA: 77884535232.04.467F64A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id E489A80192C7 for ; Fri, 5 Mar 2021 04:26:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=2mVG/6NaJ0p9YqrQ1jNJsnSFUsHzgfQvDIqsB+egAHI=; b=fiXsBiNyHiOVlScQGgeMMy82Ta RfKmP56pRubGGDoiLekMcYD2drcJzcft74CU8+sM1/RKbz4Nr6YzmeXqXLcoRHInLIbnYGHWlz68e mfv7ctNvSk+eI5ausnbEv6gqDGtG2NwQ1yfNJPF1Qi5yLtgV6GBDnZl9tohTbLJ94fMt5mWz6MPgQ Nc2bfdVG+mzIDh70LVy2wJbnKRM4goVWHF7ZJ/aeUXp6BSQQPR7UXkTlvPWjL6Ad6P9gABuRnO439 DaTZCLmfiJ0959ZU7QjLKyC+zT/Kr2MlEh2m9Ohs67rbyJIZpIOZUzv9/j0Llxa8+kA5XW25L/ChB xcWTFrcg==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI22e-00A4jU-Sh; Fri, 05 Mar 2021 04:26:27 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 22/25] mm/filemap: Convert wake_up_page_bit to wake_up_folio_bit Date: Fri, 5 Mar 2021 04:18:58 +0000 Message-Id: <20210305041901.2396498-23-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: E489A80192C7 X-Stat-Signature: pe6ouaf6be9n541keq5zw1wzccr17s3w Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918412-305717 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers have a folio, so use it directly. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 76c97cb9cbbe..e91fa14c86c7 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1094,14 +1094,14 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, return (flags & WQ_FLAG_EXCLUSIVE) != 0; } -static void wake_up_page_bit(struct page *page, int bit_nr) +static void wake_up_folio_bit(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(page); + wait_queue_head_t *q = page_waitqueue(&folio->page); struct wait_page_key key; unsigned long flags; wait_queue_entry_t bookmark; - key.page = page; + key.page = &folio->page; key.bit_nr = bit_nr; key.page_match = 0; @@ -1136,7 +1136,7 @@ static void wake_up_page_bit(struct page *page, int bit_nr) * page waiters. */ if (!waitqueue_active(q) || !key.page_match) { - ClearPageWaiters(page); + ClearFolioWaiters(folio); /* * It's possible to miss clearing Waiters here, when we woke * our page waiters, but the hashed waitqueue has waiters for @@ -1152,7 +1152,7 @@ static void wake_up_folio(struct folio *folio, int bit) { if (!FolioWaiters(folio)) return; - wake_up_page_bit(&folio->page, bit); + wake_up_folio_bit(folio, bit); } /* @@ -1417,7 +1417,7 @@ void unlock_folio(struct folio *folio) BUILD_BUG_ON(PG_waiters != 7); VM_BUG_ON_FOLIO(!FolioLocked(folio), folio); if (clear_bit_unlock_is_negative_byte(PG_locked, folio_flags(folio))) - wake_up_page_bit(&folio->page, PG_locked); + wake_up_folio_bit(folio, PG_locked); } EXPORT_SYMBOL(unlock_folio); From patchwork Fri Mar 5 04:18:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69162C433DB for ; Fri, 5 Mar 2021 04:27:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1113864FE4 for ; Fri, 5 Mar 2021 04:27:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1113864FE4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9AF116B000A; Thu, 4 Mar 2021 23:27:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9866D6B000D; Thu, 4 Mar 2021 23:27:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 828786B000E; Thu, 4 Mar 2021 23:27:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0087.hostedemail.com [216.40.44.87]) by kanga.kvack.org (Postfix) with ESMTP id 65A9B6B000A for ; Thu, 4 Mar 2021 23:27:14 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 2BD30365D for ; Fri, 5 Mar 2021 04:27:14 +0000 (UTC) X-FDA: 77884535988.11.8AB7B86 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 0960AE0011E3 for ; Fri, 5 Mar 2021 04:27:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=zWeB6aqDwj65PTFEYVpI7YPXR8x4bwFky9HOKv5MiL0=; b=LLWyhqLGjwShJrcyau7YFLyoZm JK4v5iTnEt2pgVKNSSpd28WJZebmAJvbBCSpYzRytFLx1A0yVR9j+NtlmCDCHYkRyVVmv6tCxSUwG XB4Xvv5fFCXDdDaKVXX4bJE6XQkg3incAQVqkXsgf7CtKHPeE8664Jo/uFpTbXuznhd7CaznfhbPJ 7YFQw1wO/q31+PBPaFxI0P/lm/tcKF6YQ/Yb41iVJdU9DxonpL/Qw55a0S5lIhev7sE0fW1ioqLm4 nhWmgAXLaBE/EDICf11R5An6v77/W9FrXeJSgfMljBf7yuYNLM13bkEbUK88X+cP+dz2fAtXDpAdP AG+vd6Jw==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI231-00A4pJ-TB; Fri, 05 Mar 2021 04:26:48 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 23/25] mm/page-writeback: Convert test_clear_page_writeback to take a folio Date: Fri, 5 Mar 2021 04:18:59 +0000 Message-Id: <20210305041901.2396498-24-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: fupgb6k9c3okwjcj5uw7cdibaao3cwyi X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0960AE0011E3 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918432-276554 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The one caller of test_clear_page_writeback() already has a folio, so rename it to test_clear_folio_writeback() to make it clear that it operates on the entire folio. This removes a few calls to compound_head() but actually grows the function by 49 bytes because it now accounts for the number of pages in the folio. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/page-flags.h | 2 +- mm/filemap.c | 2 +- mm/page-writeback.c | 20 ++++++++++---------- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 90381858d901..01aa4a71bf14 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -594,7 +594,7 @@ static __always_inline void SetPageUptodate(struct page *page) CLEARPAGEFLAG(Uptodate, uptodate, PF_NO_TAIL) -int test_clear_page_writeback(struct page *page); +int test_clear_folio_writeback(struct folio *folio); int __test_set_page_writeback(struct page *page, bool keep_write); #define test_set_page_writeback(page) \ diff --git a/mm/filemap.c b/mm/filemap.c index e91fa14c86c7..57f46ff2f230 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1446,7 +1446,7 @@ void end_folio_writeback(struct folio *folio) * reused before the wake_up_folio(). */ get_folio(folio); - if (!test_clear_page_writeback(&folio->page)) + if (!test_clear_folio_writeback(folio)) BUG(); smp_mb__after_atomic(); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 6c1b4737c383..fa3411ea4cd3 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -589,7 +589,7 @@ static void wb_domain_writeout_inc(struct wb_domain *dom, /* * Increment @wb's writeout completion count and the global writeout - * completion count. Called from test_clear_page_writeback(). + * completion count. Called from test_clear_folio_writeback(). */ static inline void __wb_writeout_inc(struct bdi_writeback *wb) { @@ -2719,24 +2719,24 @@ int clear_page_dirty_for_io(struct page *page) } EXPORT_SYMBOL(clear_page_dirty_for_io); -int test_clear_page_writeback(struct page *page) +int test_clear_folio_writeback(struct folio *folio) { - struct address_space *mapping = page_mapping(page); + struct address_space *mapping = folio_mapping(folio); struct mem_cgroup *memcg; struct lruvec *lruvec; int ret; - memcg = lock_page_memcg(page); - lruvec = mem_cgroup_page_lruvec(page, page_pgdat(page)); + memcg = lock_folio_memcg(folio); + lruvec = mem_cgroup_folio_lruvec(folio, folio_pgdat(folio)); if (mapping && mapping_use_writeback_tags(mapping)) { struct inode *inode = mapping->host; struct backing_dev_info *bdi = inode_to_bdi(inode); unsigned long flags; xa_lock_irqsave(&mapping->i_pages, flags); - ret = TestClearPageWriteback(page); + ret = TestClearFolioWriteback(folio); if (ret) { - __xa_clear_mark(&mapping->i_pages, page_index(page), + __xa_clear_mark(&mapping->i_pages, folio_index(folio), PAGECACHE_TAG_WRITEBACK); if (bdi->capabilities & BDI_CAP_WRITEBACK_ACCT) { struct bdi_writeback *wb = inode_to_wb(inode); @@ -2752,12 +2752,12 @@ int test_clear_page_writeback(struct page *page) xa_unlock_irqrestore(&mapping->i_pages, flags); } else { - ret = TestClearPageWriteback(page); + ret = TestClearFolioWriteback(folio); } if (ret) { dec_lruvec_state(lruvec, NR_WRITEBACK); - dec_zone_page_state(page, NR_ZONE_WRITE_PENDING); - inc_node_page_state(page, NR_WRITTEN); + dec_zone_folio_stat(folio, NR_ZONE_WRITE_PENDING); + inc_node_folio_stat(folio, NR_WRITTEN); } __unlock_page_memcg(memcg); return ret; From patchwork Fri Mar 5 04:19:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47C36C433E0 for ; Fri, 5 Mar 2021 04:28:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DD3E264FE4 for ; Fri, 5 Mar 2021 04:28:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DD3E264FE4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6EE826B000D; Thu, 4 Mar 2021 23:28:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 69E9D6B000E; Thu, 4 Mar 2021 23:28:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53EEA6B0010; Thu, 4 Mar 2021 23:28:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0241.hostedemail.com [216.40.44.241]) by kanga.kvack.org (Postfix) with ESMTP id 36FB46B000D for ; Thu, 4 Mar 2021 23:28:03 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E3EB9513 for ; Fri, 5 Mar 2021 04:28:02 +0000 (UTC) X-FDA: 77884538004.01.243C5F6 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 566FB6000104 for ; Fri, 5 Mar 2021 04:28:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xwpBIqAEuLM6Zld1KeVwy1SyI/6FWtXxGIYJ0bk67Ms=; b=nZiEQlDA3UiHbQK+LAi8BPFS61 pB+mHWU8LiaXKjDF3kxZeSTkTK3OkjP4umVg2cdiABFuLz6H45hnwPCd0a7ybRUwVsyHA1OIzANUT 5Q2d/uFqBS1raiTwf8RvPqhUP7VGa6jAU6vtBF8Za/ihsNvQdLSZLKCarFURQFzlNWGxNao0bxhCh eG6rqCE1S/m/nstsiTogOdORYksqHAXQoQHAhzbOFgXwtEvgJe40VyiAUdoL6xPLtsvnUs3MrDkNa 9y1pNasEG50jfZqfwtvdyp0yOl2S0mWayXvU4qK2tjHz8PFmnp9X1PXmxxwvSn7xOXabyy9fWkbYy APFybfzA==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI23F-00A4qX-PL; Fri, 05 Mar 2021 04:26:58 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 24/25] mm/filemap: Convert page wait queues to be folios Date: Fri, 5 Mar 2021 04:19:00 +0000 Message-Id: <20210305041901.2396498-25-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 566FB6000104 X-Stat-Signature: z75q9ngpe57oc1of3rgzopzuorp4ofbf Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918481-411075 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reinforce that if we're waiting for a bit in a struct page, that's actually in the head page by changing the type from page to folio. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 6 +++--- mm/filemap.c | 30 ++++++++++++++++-------------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 6ee4bc843f98..2236f726bf01 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -611,13 +611,13 @@ static inline pgoff_t linear_page_index(struct vm_area_struct *vma, /* This has the same layout as wait_bit_key - see fs/cachefiles/rdwr.c */ struct wait_page_key { - struct page *page; + struct folio *folio; int bit_nr; int page_match; }; struct wait_page_queue { - struct page *page; + struct folio *folio; int bit_nr; wait_queue_entry_t wait; }; @@ -625,7 +625,7 @@ struct wait_page_queue { static inline bool wake_page_match(struct wait_page_queue *wait_page, struct wait_page_key *key) { - if (wait_page->page != key->page) + if (wait_page->folio != key->folio) return false; key->page_match = 1; diff --git a/mm/filemap.c b/mm/filemap.c index 57f46ff2f230..1cdd565c69a6 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -992,11 +992,11 @@ EXPORT_SYMBOL(__page_cache_alloc); */ #define PAGE_WAIT_TABLE_BITS 8 #define PAGE_WAIT_TABLE_SIZE (1 << PAGE_WAIT_TABLE_BITS) -static wait_queue_head_t page_wait_table[PAGE_WAIT_TABLE_SIZE] __cacheline_aligned; +static wait_queue_head_t folio_wait_table[PAGE_WAIT_TABLE_SIZE] __cacheline_aligned; -static wait_queue_head_t *page_waitqueue(struct page *page) +static wait_queue_head_t *folio_waitqueue(struct folio *folio) { - return &page_wait_table[hash_ptr(page, PAGE_WAIT_TABLE_BITS)]; + return &folio_wait_table[hash_ptr(folio, PAGE_WAIT_TABLE_BITS)]; } void __init pagecache_init(void) @@ -1004,7 +1004,7 @@ void __init pagecache_init(void) int i; for (i = 0; i < PAGE_WAIT_TABLE_SIZE; i++) - init_waitqueue_head(&page_wait_table[i]); + init_waitqueue_head(&folio_wait_table[i]); page_writeback_init(); } @@ -1059,10 +1059,11 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, */ flags = wait->flags; if (flags & WQ_FLAG_EXCLUSIVE) { - if (test_bit(key->bit_nr, &key->page->flags)) + if (test_bit(key->bit_nr, &key->folio->page.flags)) return -1; if (flags & WQ_FLAG_CUSTOM) { - if (test_and_set_bit(key->bit_nr, &key->page->flags)) + if (test_and_set_bit(key->bit_nr, + &key->folio->page.flags)) return -1; flags |= WQ_FLAG_DONE; } @@ -1096,12 +1097,12 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, static void wake_up_folio_bit(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(&folio->page); + wait_queue_head_t *q = folio_waitqueue(folio); struct wait_page_key key; unsigned long flags; wait_queue_entry_t bookmark; - key.page = &folio->page; + key.folio = folio; key.bit_nr = bit_nr; key.page_match = 0; @@ -1193,7 +1194,7 @@ int sysctl_page_lock_unfairness = 5; static inline int wait_on_folio_bit_common(struct folio *folio, int bit_nr, int state, enum behavior behavior) { - wait_queue_head_t *q = page_waitqueue(&folio->page); + wait_queue_head_t *q = folio_waitqueue(folio); int unfairness = sysctl_page_lock_unfairness; struct wait_page_queue wait_page; wait_queue_entry_t *wait = &wait_page.wait; @@ -1213,7 +1214,7 @@ static inline int wait_on_folio_bit_common(struct folio *folio, int bit_nr, init_wait(wait); wait->func = wake_page_function; - wait_page.page = &folio->page; + wait_page.folio = folio; wait_page.bit_nr = bit_nr; repeat: @@ -1370,12 +1371,13 @@ int put_and_wait_on_page_locked(struct page *page, int state) */ void add_page_wait_queue(struct page *page, wait_queue_entry_t *waiter) { - wait_queue_head_t *q = page_waitqueue(page); + struct folio *folio = page_folio(page); + wait_queue_head_t *q = folio_waitqueue(folio); unsigned long flags; spin_lock_irqsave(&q->lock, flags); __add_wait_queue_entry_tail(q, waiter); - SetPageWaiters(page); + SetFolioWaiters(folio); spin_unlock_irqrestore(&q->lock, flags); } EXPORT_SYMBOL_GPL(add_page_wait_queue); @@ -1503,10 +1505,10 @@ EXPORT_SYMBOL_GPL(__lock_folio_killable); int __lock_folio_async(struct folio *folio, struct wait_page_queue *wait) { - struct wait_queue_head *q = page_waitqueue(&folio->page); + struct wait_queue_head *q = folio_waitqueue(folio); int ret = 0; - wait->page = &folio->page; + wait->folio = folio; wait->bit_nr = PG_locked; spin_lock_irq(&q->lock); From patchwork Fri Mar 5 04:19:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12117607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 481B1C433E0 for ; Fri, 5 Mar 2021 04:28:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D7F0B6500C for ; Fri, 5 Mar 2021 04:28:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D7F0B6500C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 678376B0008; Thu, 4 Mar 2021 23:28:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 62A806B000A; Thu, 4 Mar 2021 23:28:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 518046B000C; Thu, 4 Mar 2021 23:28:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id 35A346B0008 for ; Thu, 4 Mar 2021 23:28:30 -0500 (EST) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id EB1068249980 for ; Fri, 5 Mar 2021 04:28:29 +0000 (UTC) X-FDA: 77884539138.03.0CD881A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf25.hostedemail.com (Postfix) with ESMTP id 54BE36000103 for ; Fri, 5 Mar 2021 04:28:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+gAkfU1JkXjwF3GYBjHXsSgNo4U4uzwhoammc6XTTH4=; b=tPyoTVAHCI/oKld3COtQHSG4d5 jTUi/S4CcVcRjWU0x1gYM8xrxwsLTVT3/D4sW8GHjZ1A3Us5Fq9MZY4Cu2k5okWptNfzXy5oWsOPR 9K0vHa1UNHJXnV8tAwYmZJd01ZD5KDiT/v5iR3oX5hfxCsz56FYJZl7p2iZfTnSHYcud3asMJeFRU FimJlvT3uCpOOkFUun3gRAHsRHU12BSgPyWyZJ0P0+ws/D5zqoZdXdvJlRrSofvJg549zYJVUxOzg ft4ulMBjCX4tM0Drdkj+EIQEvU5JZQpUQNDSDNWnVgIvm3l8FaMwOM9knEU8LWYUTB947bEusnOI9 ljeTaWBw==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI23Y-00A5SZ-2K; Fri, 05 Mar 2021 04:27:29 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 25/25] cachefiles: Switch to wait_page_key Date: Fri, 5 Mar 2021 04:19:01 +0000 Message-Id: <20210305041901.2396498-26-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: ft5wuxgkq9zbgkr6xm9kxwz61bwzgbxe X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 54BE36000103 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918508-5191 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Cachefiles was relying on wait_page_key and wait_bit_key being the same layout, which is fragile. Now that wait_page_key is exposed in the pagemap.h header, we can remove that fragility. Also switch it to use the folio directly instead of the page. Signed-off-by: Matthew Wilcox (Oracle) --- fs/cachefiles/rdwr.c | 13 ++++++------- include/linux/pagemap.h | 1 - 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c index e027c718ca01..b1dbc484a9c7 100644 --- a/fs/cachefiles/rdwr.c +++ b/fs/cachefiles/rdwr.c @@ -24,22 +24,21 @@ static int cachefiles_read_waiter(wait_queue_entry_t *wait, unsigned mode, container_of(wait, struct cachefiles_one_read, monitor); struct cachefiles_object *object; struct fscache_retrieval *op = monitor->op; - struct wait_bit_key *key = _key; - struct page *page = wait->private; + struct wait_page_key *key = _key; + struct folio *folio = wait->private; ASSERT(key); _enter("{%lu},%u,%d,{%p,%u}", monitor->netfs_page->index, mode, sync, - key->flags, key->bit_nr); + key->folio, key->bit_nr); - if (key->flags != &page->flags || - key->bit_nr != PG_locked) + if (key->folio != folio || key->bit_nr != PG_locked) return 0; - _debug("--- monitor %p %lx ---", page, page->flags); + _debug("--- monitor %p %lx ---", folio, folio->page.flags); - if (!PageUptodate(page) && !PageError(page)) { + if (!FolioUptodate(folio) && !FolioError(folio)) { /* unlocked, not uptodate and not erronous? */ _debug("page probably truncated"); } diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 2236f726bf01..699a6a0bcc54 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -609,7 +609,6 @@ static inline pgoff_t linear_page_index(struct vm_area_struct *vma, return pgoff; } -/* This has the same layout as wait_bit_key - see fs/cachefiles/rdwr.c */ struct wait_page_key { struct folio *folio; int bit_nr;