From patchwork Fri Apr 9 18:50:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF57DC433B4 for ; Fri, 9 Apr 2021 18:52:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7E5766113A for ; Fri, 9 Apr 2021 18:52:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7E5766113A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E29FD6B006E; Fri, 9 Apr 2021 14:52:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DB31E6B0070; Fri, 9 Apr 2021 14:52:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB92F6B0071; Fri, 9 Apr 2021 14:52:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0165.hostedemail.com [216.40.44.165]) by kanga.kvack.org (Postfix) with ESMTP id 971EC6B006E for ; Fri, 9 Apr 2021 14:52:34 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5A1CEBEF2 for ; Fri, 9 Apr 2021 18:52:34 +0000 (UTC) X-FDA: 78013724628.12.CAFE6DF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id CB1E9A000145 for ; Fri, 9 Apr 2021 18:52:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=k9UQJrroA7+b5FsVn9SgeyEPUnOcRSZz/F33X7W870c=; b=emxCeTdgjw5mMIOnmcPfDWAMPj coxh6Q90VA8KO3R63jZ+MYkpUByZBFAzLQG53eLmuxPvmU03KdwerAgLkZEtfNLlCCCSRC+i6RtDj hdpDnn8EJ6m7XeVoGxtM2maZRKFHY4QBb+wz9jjfEOOt8MKv1yO8ShOFVezPb5Y/xtpzCRHkfEzoR SKwX/trgOG1l6OAif3RRHAql0D1LI3hKlGPSuPW0f+yBny6g0gXCU104zvRilfepSVMaP48iehlxm 0OPCjxQjjltLtLqlYAXgQ5bIightYT4JbGazmQA4lFzThe3/PrKqkEdP/Kymlpw3YspI6LcWHjtcJ KeJZXOpg==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwEA-000n0R-Q0; Fri, 09 Apr 2021 18:51:40 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org Subject: [PATCH v7 01/28] mm: Optimise nth_page for contiguous memmap Date: Fri, 9 Apr 2021 19:50:38 +0100 Message-Id: <20210409185105.188284-2-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CB1E9A000145 X-Stat-Signature: s79szzsfze5ioio3txgw9uataey1a9yd Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617994353-127303 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the memmap is virtually contiguous (either because we're using a virtually mapped memmap or because we don't support a discontig memmap at all), then we can implement nth_page() by simple addition. Contrary to popular belief, the compiler is not able to optimise this itself for a vmemmap configuration. This reduces one example user (sg.c) by four instructions: struct page *page = nth_page(rsv_schp->pages[k], offset >> PAGE_SHIFT); before: 49 8b 45 70 mov 0x70(%r13),%rax 48 63 c9 movslq %ecx,%rcx 48 c1 eb 0c shr $0xc,%rbx 48 8b 04 c8 mov (%rax,%rcx,8),%rax 48 2b 05 00 00 00 00 sub 0x0(%rip),%rax R_X86_64_PC32 vmemmap_base-0x4 48 c1 f8 06 sar $0x6,%rax 48 01 d8 add %rbx,%rax 48 c1 e0 06 shl $0x6,%rax 48 03 05 00 00 00 00 add 0x0(%rip),%rax R_X86_64_PC32 vmemmap_base-0x4 after: 49 8b 45 70 mov 0x70(%r13),%rax 48 63 c9 movslq %ecx,%rcx 48 c1 eb 0c shr $0xc,%rbx 48 c1 e3 06 shl $0x6,%rbx 48 03 1c c8 add (%rax,%rcx,8),%rbx Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- include/linux/mm.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index b58c73e50da0..036f63a44a5c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -234,7 +234,11 @@ int overcommit_policy_handler(struct ctl_table *, int, void *, size_t *, int __add_to_page_cache_locked(struct page *page, struct address_space *mapping, pgoff_t index, gfp_t gfp, void **shadowp); +#if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP) #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n)) +#else +#define nth_page(page,n) ((page) + (n)) +#endif /* to align the pointer to the (next) page boundary */ #define PAGE_ALIGN(addr) ALIGN(addr, PAGE_SIZE) From patchwork Fri Apr 9 18:50:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91556C43460 for ; Fri, 9 Apr 2021 18:55:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0B0AB61106 for ; Fri, 9 Apr 2021 18:55:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B0AB61106 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8A7E96B006C; Fri, 9 Apr 2021 14:55:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8580E6B006E; Fri, 9 Apr 2021 14:55:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D18A6B0070; Fri, 9 Apr 2021 14:55:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0095.hostedemail.com [216.40.44.95]) by kanga.kvack.org (Postfix) with ESMTP id 4EE666B006C for ; Fri, 9 Apr 2021 14:55:00 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0A13F18458A8A for ; Fri, 9 Apr 2021 18:55:00 +0000 (UTC) X-FDA: 78013730760.22.65DD1A1 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 77965E000111 for ; Fri, 9 Apr 2021 18:54:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=r90mtSioPB9UcKNT6Rnzzil4xndwmqmNbGk9sDR1/6M=; b=cZiKuQoyvvEdpoA3aWQU65R+Ry g2UJvVW2QGxdoJW45OmJDc+LqOZG8C0Edq4xcE/1Z2z8SqpoQnOfPDJUIij8GWEHAIonZXkEzIt5E pT2fo9ms8107sBFhHHd2jUVWrF7/zLrr7B0uBvrvW3HXmsYD7ChEO4DaH0Yfs5kBvQIxY+oU3lXzs QnYzWtDzLV4OeScMWg+LMpoKMjJe7C/MxQoa8zmMAhEx2lPNBPrsd+auld1y0jyLyaAgkXBK2q3Qz kQIlWETnAOx9+vf3i2wDVyzvkrKwcl/Z4oE29cvd5zXaOofe1c6PCFN5saYyWtQYlYKlyvNBouAFC plFbHyJg==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwG0-000n8t-Pm; Fri, 09 Apr 2021 18:53:35 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Christoph Hellwig , Jeff Layton Subject: [PATCH v7 04/28] mm/vmstat: Add functions to account folio statistics Date: Fri, 9 Apr 2021 19:50:41 +0100 Message-Id: <20210409185105.188284-5-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 77965E000111 X-Stat-Signature: 1nubyq8nonsycmru3oyoimb54n1shgax X-Rspamd-Server: rspam02 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617994498-470201 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow page counters to be more readily modified by callers which have a folio. Name these wrappers with 'stat' instead of 'state' as requested by Linus here: https://lore.kernel.org/linux-mm/CAHk-=wj847SudR-kt+46fT3+xFFgiwpgThvm7DJWGdi4cVrbnQ@mail.gmail.com/ Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- include/linux/vmstat.h | 107 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 3299cd69e4ca..d287d7c31b8f 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -402,6 +402,78 @@ static inline void drain_zonestat(struct zone *zone, struct per_cpu_pageset *pset) { } #endif /* CONFIG_SMP */ +static inline void __zone_stat_mod_folio(struct folio *folio, + enum zone_stat_item item, long nr) +{ + __mod_zone_page_state(folio_zone(folio), item, nr); +} + +static inline void __zone_stat_add_folio(struct folio *folio, + enum zone_stat_item item) +{ + __mod_zone_page_state(folio_zone(folio), item, folio_nr_pages(folio)); +} + +static inline void __zone_stat_sub_folio(struct folio *folio, + enum zone_stat_item item) +{ + __mod_zone_page_state(folio_zone(folio), item, -folio_nr_pages(folio)); +} + +static inline void zone_stat_mod_folio(struct folio *folio, + enum zone_stat_item item, long nr) +{ + mod_zone_page_state(folio_zone(folio), item, nr); +} + +static inline void zone_stat_add_folio(struct folio *folio, + enum zone_stat_item item) +{ + mod_zone_page_state(folio_zone(folio), item, folio_nr_pages(folio)); +} + +static inline void zone_stat_sub_folio(struct folio *folio, + enum zone_stat_item item) +{ + mod_zone_page_state(folio_zone(folio), item, -folio_nr_pages(folio)); +} + +static inline void __node_stat_mod_folio(struct folio *folio, + enum node_stat_item item, long nr) +{ + __mod_node_page_state(folio_pgdat(folio), item, nr); +} + +static inline void __node_stat_add_folio(struct folio *folio, + enum node_stat_item item) +{ + __mod_node_page_state(folio_pgdat(folio), item, folio_nr_pages(folio)); +} + +static inline void __node_stat_sub_folio(struct folio *folio, + enum node_stat_item item) +{ + __mod_node_page_state(folio_pgdat(folio), item, -folio_nr_pages(folio)); +} + +static inline void node_stat_mod_folio(struct folio *folio, + enum node_stat_item item, long nr) +{ + mod_node_page_state(folio_pgdat(folio), item, nr); +} + +static inline void node_stat_add_folio(struct folio *folio, + enum node_stat_item item) +{ + mod_node_page_state(folio_pgdat(folio), item, folio_nr_pages(folio)); +} + +static inline void node_stat_sub_folio(struct folio *folio, + enum node_stat_item item) +{ + mod_node_page_state(folio_pgdat(folio), item, -folio_nr_pages(folio)); +} + static inline void __mod_zone_freepage_state(struct zone *zone, int nr_pages, int migratetype) { @@ -530,6 +602,24 @@ static inline void __dec_lruvec_page_state(struct page *page, __mod_lruvec_page_state(page, idx, -1); } +static inline void __lruvec_stat_mod_folio(struct folio *folio, + enum node_stat_item idx, int val) +{ + __mod_lruvec_page_state(&folio->page, idx, val); +} + +static inline void __lruvec_stat_add_folio(struct folio *folio, + enum node_stat_item idx) +{ + __lruvec_stat_mod_folio(folio, idx, folio_nr_pages(folio)); +} + +static inline void __lruvec_stat_sub_folio(struct folio *folio, + enum node_stat_item idx) +{ + __lruvec_stat_mod_folio(folio, idx, -folio_nr_pages(folio)); +} + static inline void inc_lruvec_page_state(struct page *page, enum node_stat_item idx) { @@ -542,4 +632,21 @@ static inline void dec_lruvec_page_state(struct page *page, mod_lruvec_page_state(page, idx, -1); } +static inline void lruvec_stat_mod_folio(struct folio *folio, + enum node_stat_item idx, int val) +{ + mod_lruvec_page_state(&folio->page, idx, val); +} + +static inline void lruvec_stat_add_folio(struct folio *folio, + enum node_stat_item idx) +{ + lruvec_stat_mod_folio(folio, idx, folio_nr_pages(folio)); +} + +static inline void lruvec_stat_sub_folio(struct folio *folio, + enum node_stat_item idx) +{ + lruvec_stat_mod_folio(folio, idx, -folio_nr_pages(folio)); +} #endif /* _LINUX_VMSTAT_H */ From patchwork Fri Apr 9 18:50:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C831EC433B4 for ; Fri, 9 Apr 2021 18:55:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6D08661106 for ; Fri, 9 Apr 2021 18:55:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6D08661106 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EC2246B006E; Fri, 9 Apr 2021 14:55:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E990E6B0070; Fri, 9 Apr 2021 14:55:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D61116B0071; Fri, 9 Apr 2021 14:55:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0219.hostedemail.com [216.40.44.219]) by kanga.kvack.org (Postfix) with ESMTP id BD0096B006E for ; Fri, 9 Apr 2021 14:55:47 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 6C1E48248047 for ; Fri, 9 Apr 2021 18:55:47 +0000 (UTC) X-FDA: 78013732734.40.CB6C924 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id BF1F4A00014C for ; Fri, 9 Apr 2021 18:55:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=WgtJerUq2+EOZX8C2WOKla9RTZ8JhXvu2XoZvGKvg78=; b=e3HQD3msEJHMxXsmkkrKNF4sXX KDhDNJJhOfUj+79VS1ymu8qo/BSfNF0bk9pFrJA1BZhoFqUgSpkHq+sh/v07XkgpXvzj22LOV7DYr cIH/LxpZh81SrLoHDjWRHN8PJROHJxeYjajTehd0CrTMiCXM7B/pIeCPUz2HAItrsHrQ8ofSQVK7f ksoBWLVyzSZBOTSWZ6Ef/Amt9uxshjJPtjayWZvg/o546B+zctA4yXHuCDsO3ObmgqC1F1kVIwrsX I0+LM6X8W9bcTonZeyGlq8qnPXhXqWNLwNuIrArJIqSGtrnyDsvAiCEH0XRUkH2B/0MdR7X+R8Ugb 151sqtBw==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwGp-000nDv-NO; Fri, 09 Apr 2021 18:54:37 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Zi Yan , Christoph Hellwig , Jeff Layton Subject: [PATCH v7 05/28] mm/debug: Add VM_BUG_ON_FOLIO and VM_WARN_ON_ONCE_FOLIO Date: Fri, 9 Apr 2021 19:50:42 +0100 Message-Id: <20210409185105.188284-6-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: BF1F4A00014C X-Stat-Signature: tsnnimzkaj1b1xrhghsjau3p1bx8gx39 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617994546-275475 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These are the folio equivalents of VM_BUG_ON_PAGE and VM_WARN_ON_ONCE_PAGE. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- include/linux/mmdebug.h | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/include/linux/mmdebug.h b/include/linux/mmdebug.h index 5d0767cb424a..77d24e1dcaec 100644 --- a/include/linux/mmdebug.h +++ b/include/linux/mmdebug.h @@ -23,6 +23,13 @@ void dump_mm(const struct mm_struct *mm); BUG(); \ } \ } while (0) +#define VM_BUG_ON_FOLIO(cond, folio) \ + do { \ + if (unlikely(cond)) { \ + dump_page(&folio->page, "VM_BUG_ON_FOLIO(" __stringify(cond)")");\ + BUG(); \ + } \ + } while (0) #define VM_BUG_ON_VMA(cond, vma) \ do { \ if (unlikely(cond)) { \ @@ -48,6 +55,17 @@ void dump_mm(const struct mm_struct *mm); } \ unlikely(__ret_warn_once); \ }) +#define VM_WARN_ON_ONCE_FOLIO(cond, folio) ({ \ + static bool __section(".data.once") __warned; \ + int __ret_warn_once = !!(cond); \ + \ + if (unlikely(__ret_warn_once && !__warned)) { \ + dump_page(&folio->page, "VM_WARN_ON_ONCE_FOLIO(" __stringify(cond)")");\ + __warned = true; \ + WARN_ON(1); \ + } \ + unlikely(__ret_warn_once); \ +}) #define VM_WARN_ON(cond) (void)WARN_ON(cond) #define VM_WARN_ON_ONCE(cond) (void)WARN_ON_ONCE(cond) @@ -56,11 +74,13 @@ void dump_mm(const struct mm_struct *mm); #else #define VM_BUG_ON(cond) BUILD_BUG_ON_INVALID(cond) #define VM_BUG_ON_PAGE(cond, page) VM_BUG_ON(cond) +#define VM_BUG_ON_FOLIO(cond, folio) VM_BUG_ON(cond) #define VM_BUG_ON_VMA(cond, vma) VM_BUG_ON(cond) #define VM_BUG_ON_MM(cond, mm) VM_BUG_ON(cond) #define VM_WARN_ON(cond) BUILD_BUG_ON_INVALID(cond) #define VM_WARN_ON_ONCE(cond) BUILD_BUG_ON_INVALID(cond) #define VM_WARN_ON_ONCE_PAGE(cond, page) BUILD_BUG_ON_INVALID(cond) +#define VM_WARN_ON_ONCE_FOLIO(cond, folio) BUILD_BUG_ON_INVALID(cond) #define VM_WARN_ONCE(cond, format...) BUILD_BUG_ON_INVALID(cond) #define VM_WARN(cond, format...) BUILD_BUG_ON_INVALID(cond) #endif From patchwork Fri Apr 9 18:50:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D80BC433ED for ; Fri, 9 Apr 2021 18:57:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E356661106 for ; Fri, 9 Apr 2021 18:57:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E356661106 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7BB946B006E; Fri, 9 Apr 2021 14:57:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 792726B0070; Fri, 9 Apr 2021 14:57:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 659FF6B0071; Fri, 9 Apr 2021 14:57:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0054.hostedemail.com [216.40.44.54]) by kanga.kvack.org (Postfix) with ESMTP id 4C55E6B006E for ; Fri, 9 Apr 2021 14:57:30 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id E9B17184B5000 for ; Fri, 9 Apr 2021 18:57:29 +0000 (UTC) X-FDA: 78013737018.02.4967CE3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf12.hostedemail.com (Postfix) with ESMTP id D2435D2 for ; Fri, 9 Apr 2021 18:57:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=yiZtKKoqkRI3gZZhFYuXMkcbvraC4YjINsQdfKzNzag=; b=OYYJQ++Z9BGuizodyzOFqEofq7 ADpXyQa8Dq/MVnymVRgf1QYCxfTnoC+6em1KG+bWC6qkCdfhPz3PcpFdzJUEB5pcYesxlUIj3491U 5Zow+0ExjUg6QUWAKUDokf+bklhnwgGaRyIuV5xXkZY7P+I6XZFth0pygjhlXX25lfBHQBdlqf6LB lwRinNRMU+fpvj93AlYI8gp/xlGtIUH7NSyLzY+B9t4FxT3haIoSJTWYfMFRBawxfnCxM1vtNqSGt 4wVrb2MAqIex8y3JkL8Btjc5r7qNgQO8bYwixeUPknVFKyVNzS29tfjsprONDs9bIP63i3W30HPcT Tee9rakw==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwIQ-000nQQ-AE; Fri, 09 Apr 2021 18:56:03 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Zi Yan , Christoph Hellwig , Jeff Layton Subject: [PATCH v7 07/28] mm: Add put_folio Date: Fri, 9 Apr 2021 19:50:44 +0100 Message-Id: <20210409185105.188284-8-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: xid1j8zzhqeuact11aa9jsg4eiytuqf4 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D2435D2 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf12; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617994645-304853 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we know we have a folio, we can call put_folio() instead of put_page() and save the overhead of calling compound_head(). Also skips the devmap checks. This commit looks like it should be a no-op, but actually saves 1312 bytes of text with the distro-derived config that I'm testing. Some functions grow a little while others shrink. I presume the compiler is making different inlining decisions. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- include/linux/mm.h | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 4c98b52613b7..747c6f47aef6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -751,6 +751,11 @@ static inline int put_page_testzero(struct page *page) return page_ref_dec_and_test(page); } +static inline int put_folio_testzero(struct folio *folio) +{ + return put_page_testzero(&folio->page); +} + /* * Try to grab a ref unless the page has a refcount of zero, return false if * that is the case. @@ -1242,9 +1247,28 @@ static inline __must_check bool try_get_page(struct page *page) return true; } +/** + * put_folio - Decrement the reference count on a folio. + * @folio: The folio. + * + * If the folio's reference count reaches zero, the memory will be + * released back to the page allocator and may be used by another + * allocation immediately. Do not access the memory or the struct folio + * after calling put_folio() unless you can be sure that it wasn't the + * last reference. + * + * Context: May be called in process or interrupt context, but not in NMI + * context. May be called while holding a spinlock. + */ +static inline void put_folio(struct folio *folio) +{ + if (put_folio_testzero(folio)) + __put_page(&folio->page); +} + static inline void put_page(struct page *page) { - page = compound_head(page); + struct folio *folio = page_folio(page); /* * For devmap managed pages we need to catch refcount transition from @@ -1252,13 +1276,12 @@ static inline void put_page(struct page *page) * need to inform the device driver through callback. See * include/linux/memremap.h and HMM for details. */ - if (page_is_devmap_managed(page)) { - put_devmap_managed_page(page); + if (page_is_devmap_managed(&folio->page)) { + put_devmap_managed_page(&folio->page); return; } - if (put_page_testzero(page)) - __put_page(page); + put_folio(folio); } /* From patchwork Fri Apr 9 18:50:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6D947C433B4 for ; Fri, 9 Apr 2021 18:57:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CADE16115B for ; Fri, 9 Apr 2021 18:57:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CADE16115B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 65B786B006C; Fri, 9 Apr 2021 14:57:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 632836B0070; Fri, 9 Apr 2021 14:57:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FAA36B0071; Fri, 9 Apr 2021 14:57:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 365C46B006C for ; Fri, 9 Apr 2021 14:57:50 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D784C8248047 for ; Fri, 9 Apr 2021 18:57:49 +0000 (UTC) X-FDA: 78013737858.21.12B307E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id E3CE190009ED for ; Fri, 9 Apr 2021 18:57:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=W5g+VwHltNTj8FpdbjCM8nUwZBIK09QAEf3IQR2EfkI=; b=FJcUD5OWaQ1F07avMpHBfY5M8P Dq9vUamNO9uOBi0OqHdFa5M2xUIlujch1gP4Sq18agRZ4fX6IET6hmXf5j6GgyKsQjUR8IKsxMd5t EaDtTeyd6YmXiw3NF9S+LxAwEPWD/AL+08HNy+h6+BxsoT+MnqnTqTeqIQ63KsR+f5ZlDdkdInC/2 70NEC6n+CGkiLiUASJbTwLcvZ7IVLlY+YOM+tGQ+6TJ3hNXjPyNeuoIJJKPyn1fRCG/1r3OnTIZ9Y gT1c1jbBo8HsZVPKkJKQg8OtRsTEVFx+Z6NWteWLjNbmuWV60+Sfa2510Gievcvh2M9mWVeOlBt3d ljYiYDFA==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwJ1-000nZl-VF; Fri, 09 Apr 2021 18:56:52 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Zi Yan , Christoph Hellwig , Jeff Layton Subject: [PATCH v7 08/28] mm: Add get_folio Date: Fri, 9 Apr 2021 19:50:45 +0100 Message-Id: <20210409185105.188284-9-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: csaqjuygpx7srtpxzadbkjjonp4fjid1 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E3CE190009ED Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617994658-403429 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If we know we have a folio, we can call get_folio() instead of get_page() and save the overhead of calling compound_head(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- include/linux/mm.h | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 747c6f47aef6..67d9104c1cc1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1219,18 +1219,26 @@ static inline bool is_pci_p2pdma_page(const struct page *page) } /* 127: arbitrary random number, small enough to assemble well */ -#define page_ref_zero_or_close_to_overflow(page) \ - ((unsigned int) page_ref_count(page) + 127u <= 127u) +#define folio_ref_zero_or_close_to_overflow(folio) \ + ((unsigned int) folio_ref_count(folio) + 127u <= 127u) + +/** + * get_folio - Increment the reference count on a folio. + * @folio: The folio. + * + * Context: May be called in any context, as long as you know that + * you have a refcount on the folio. If you do not already have one, + * try_grab_page() may be the right interface for you to use. + */ +static inline void get_folio(struct folio *folio) +{ + VM_BUG_ON_FOLIO(folio_ref_zero_or_close_to_overflow(folio), folio); + folio_ref_inc(folio); +} static inline void get_page(struct page *page) { - page = compound_head(page); - /* - * Getting a normal page or the head of a compound page - * requires to already have an elevated page->_refcount. - */ - VM_BUG_ON_PAGE(page_ref_zero_or_close_to_overflow(page), page); - page_ref_inc(page); + get_folio(page_folio(page)); } bool __must_check try_grab_page(struct page *page, unsigned int flags); From patchwork Fri Apr 9 18:50:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D235DC433ED for ; Fri, 9 Apr 2021 19:01:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6A6776115C for ; Fri, 9 Apr 2021 19:01:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6A6776115C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DF8C76B006E; Fri, 9 Apr 2021 15:01:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D820C6B0070; Fri, 9 Apr 2021 15:01:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B856D6B0071; Fri, 9 Apr 2021 15:01:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0178.hostedemail.com [216.40.44.178]) by kanga.kvack.org (Postfix) with ESMTP id 98DD46B006E for ; Fri, 9 Apr 2021 15:01:09 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5136D612E for ; Fri, 9 Apr 2021 19:01:09 +0000 (UTC) X-FDA: 78013746258.09.B14CCFC Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf06.hostedemail.com (Postfix) with ESMTP id 4EA8FC0007DA for ; Fri, 9 Apr 2021 19:00:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=bXioRnHFjaOls7rNu6Os2u7zEhcPe7aVviL+hJtS1ts=; b=Xv73+XnmqZyjls63Tkh7gDKL3I JJUqizitHPWifnJJbqMewwF3WW5mtw9wR4petzC9B8LhfRj/k1Ggo1cwxq8nC7G1IwE0Rha880hKG 2FSR06wXdf2tuNn5hld/JBrTiqdTuQT98BrgAfBQ8zUs8QlxfRAMGz6+p3bOa/yG77YTnEL2eSfjP 3VzURVZtuT1D381PM5gt0J3xXQBU8WUVyc27axbhnRRfhV+D0cVrSv2MwDQMuskZLZmsyQJjtv4VQ KSvPvjv4o6WdJQyvsW1itmLf18JwTty5SYEwc8JMpSe2cln6EHFi6d73EuRcppvuidhQO5s5DZJaS nHCDQjPg==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwLD-000niv-EU; Fri, 09 Apr 2021 18:59:04 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Christoph Hellwig , Jeff Layton Subject: [PATCH v7 11/28] mm/filemap: Add folio_index, folio_file_page and folio_contains Date: Fri, 9 Apr 2021 19:50:48 +0100 Message-Id: <20210409185105.188284-12-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: yecyart68bxzfpnkae7ea7yts45eodgj X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4EA8FC0007DA Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf06; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617994810-833489 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: folio_index() is the equivalent of page_index() for folios. folio_file_page() is the equivalent of find_subpage(). folio_contains() is the equivalent of thp_contains(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- include/linux/pagemap.h | 53 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index f2fd0b811c1b..a8b108a9ac6e 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -462,6 +462,59 @@ static inline bool thp_contains(struct page *head, pgoff_t index) return page_index(head) == (index & ~(thp_nr_pages(head) - 1UL)); } +#define swapcache_index(folio) __page_file_index(&(folio)->page) + +/** + * folio_index - File index of a folio. + * @folio: The folio. + * + * For a folio which is either in the page cache or the swap cache, + * return its index within the address_space it belongs to. If you know + * the page is definitely in the page cache, you can look at the folio's + * index directly. + * + * Return: The index (offset in units of pages) of a folio in its file. + */ +static inline pgoff_t folio_index(struct folio *folio) +{ + if (unlikely(FolioSwapCache(folio))) + return swapcache_index(folio); + return folio->index; +} + +/** + * folio_file_page - The page for a particular index. + * @folio: The folio which contains this index. + * @index: The index we want to look up. + * + * Sometimes after looking up a folio in the page cache, we need to + * obtain the specific page for an index (eg a page fault). + * + * Return: The page containing the file data for this index. + */ +static inline struct page *folio_file_page(struct folio *folio, pgoff_t index) +{ + return nth_page(&folio->page, index & (folio_nr_pages(folio) - 1)); +} + +/** + * folio_contains - Does this folio contain this index? + * @folio: The folio. + * @index: The page index within the file. + * + * Context: The caller should have the page locked in order to prevent + * (eg) shmem from moving the page between the page cache and swap cache + * and changing its index in the middle of the operation. + * Return: true or false. + */ +static inline bool folio_contains(struct folio *folio, pgoff_t index) +{ + /* HugeTLBfs indexes the page cache in units of hpage_size */ + if (PageHuge(&folio->page)) + return folio->index == index; + return index - folio_index(folio) < folio_nr_pages(folio); +} + /* * Given the page we found in the page cache, return the page corresponding * to this index in the file From patchwork Fri Apr 9 18:50:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F330C433B4 for ; Fri, 9 Apr 2021 19:00:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A90E761106 for ; Fri, 9 Apr 2021 19:00:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A90E761106 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 448F76B006C; Fri, 9 Apr 2021 15:00:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E5E66B006E; Fri, 9 Apr 2021 15:00:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 289206B0070; Fri, 9 Apr 2021 15:00:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0157.hostedemail.com [216.40.44.157]) by kanga.kvack.org (Postfix) with ESMTP id 0800F6B006C for ; Fri, 9 Apr 2021 15:00:43 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BC4B8365F for ; Fri, 9 Apr 2021 19:00:42 +0000 (UTC) X-FDA: 78013745124.12.699C386 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id D46A7801C7D8 for ; Fri, 9 Apr 2021 19:00:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Jy8OgE/ibRhYpmxJWqJ5F9J87NOfXTTmiZFJczWvAp4=; b=BwgYB9MBM7oWYpGFyfrTwgPYE7 ZSfW3lcXj4cSu2aNL0y8k4aEem7psK5ME7uwysts4LogpBrG3VvT4d4L7T1V60cRxyh31lNPbm/v7 ocqrpv4eMepzMZ1h8b37Ohpcm1QXhWvNNjkQuHX0Eo7nWlT4eC8Tn/TYfsoQbYvu1DuoMuFWohSBy mKDDEfrInd0kzeEw0XyVPy3HHITvgLDFzix0kr2+2IBFl6EYChQLztkXZJ873GEWGh2mvjlZMvzF+ 3eIDQcAHi7crkwY0u/ue4DowFsMVpaJ6g2Xoz+fNXbd76qvlYhU6sjY3eZJC3xuIH4fSHDehmay1o nC3U8MKQ==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwLn-000nlx-Ee; Fri, 09 Apr 2021 18:59:42 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Christoph Hellwig , Jeff Layton Subject: [PATCH v7 12/28] mm/filemap: Add folio_next_index Date: Fri, 9 Apr 2021 19:50:49 +0100 Message-Id: <20210409185105.188284-13-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: 9mh1wacuuam5m1dq5f9bnzx6am9gkamo X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D46A7801C7D8 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617994841-894016 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This helper returns the page index of the next folio in the file (ie the end of this folio, plus one). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- include/linux/pagemap.h | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index a8b108a9ac6e..5130503519b0 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -482,6 +482,17 @@ static inline pgoff_t folio_index(struct folio *folio) return folio->index; } +/** + * folio_next_index - Get the index of the next folio. + * @folio: The current folio. + * + * Return: The index of the folio which follows this folio in the file. + */ +static inline pgoff_t folio_next_index(struct folio *folio) +{ + return folio->index + folio_nr_pages(folio); +} + /** * folio_file_page - The page for a particular index. * @folio: The folio which contains this index. From patchwork Fri Apr 9 18:50:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D1F3C433B4 for ; Fri, 9 Apr 2021 19:04:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DF60D6115C for ; Fri, 9 Apr 2021 19:04:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DF60D6115C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5B8726B006C; Fri, 9 Apr 2021 15:04:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 58ECE6B006E; Fri, 9 Apr 2021 15:04:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 47DA76B0070; Fri, 9 Apr 2021 15:04:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0144.hostedemail.com [216.40.44.144]) by kanga.kvack.org (Postfix) with ESMTP id 2B8FA6B006C for ; Fri, 9 Apr 2021 15:04:04 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id DEC6618057FE4 for ; Fri, 9 Apr 2021 19:04:03 +0000 (UTC) X-FDA: 78013753566.39.2654EC9 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf02.hostedemail.com (Postfix) with ESMTP id B8D7340002C6 for ; Fri, 9 Apr 2021 19:03:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Mf5he/ba/r+UNHVPyU98btIvka6mXa95Hf7mjCLcMVg=; b=MkIcROPHjS314CkgaiIP1CwgF5 0HmmGbBqvwsE50d2qgjUX3MrkVflXJ19XWFcijM7SFcDtqGhyYBIK0c8B4Isau+a9cqhhfWjdhRbg 8L4ixVGGE4FU60SyPRkUHx5HpAIgad3XMG3+9ps6P9wyWpxJnkn9NmlwAn9FeTlrV7PR2d5wT94Lg 1b/D/5+nRwq3f+SKSu/eRu/PUy7mAAk9Rvpk1VORM/9KXDWSTL3OII/91qH9b2wvD1WM+x8f6888p Izb2bbn81A5bzHbxyzJCjAH52N7a0Yl0JsFeJAhz2Q8thoXIu1QyCr4/UbhW+onpHBFlLoN3Ics9Y 3KGAI4bQ==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwOC-000o2d-SM; Fri, 09 Apr 2021 19:02:25 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Christoph Hellwig , Jeff Layton Subject: [PATCH v7 15/28] mm: Add folio_mapcount Date: Fri, 9 Apr 2021 19:50:52 +0100 Message-Id: <20210409185105.188284-16-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: kpb1suyf4xrpk3bg6m6htjkuwox5o4si X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B8D7340002C6 Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf02; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617995032-84368 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is the folio equivalent of page_mapcount(). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- include/linux/mm.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 143b354c3f4a..7bd2ce197e2f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -883,6 +883,22 @@ static inline int page_mapcount(struct page *page) return atomic_read(&page->_mapcount) + 1; } +/** + * folio_mapcount - The number of mappings of this folio. + * @folio: The folio. + * + * The result includes the number of times any of the pages in the + * folio are mapped to userspace. + * + * Return: The number of page table entries which refer to this folio. + */ +static inline int folio_mapcount(struct folio *folio) +{ + if (unlikely(FolioMulti(folio))) + return __page_mapcount(&folio->page); + return atomic_read(&folio->_mapcount) + 1; +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE int total_mapcount(struct page *page); int page_trans_huge_mapcount(struct page *page, int *total_mapcount); From patchwork Fri Apr 9 18:50:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4235AC433ED for ; Fri, 9 Apr 2021 19:04:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D594A6115C for ; Fri, 9 Apr 2021 19:04:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D594A6115C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 716D96B006E; Fri, 9 Apr 2021 15:04:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EDF96B0070; Fri, 9 Apr 2021 15:04:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B59A6B0071; Fri, 9 Apr 2021 15:04:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id 41D526B006E for ; Fri, 9 Apr 2021 15:04:52 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0C183A8DD for ; Fri, 9 Apr 2021 19:04:52 +0000 (UTC) X-FDA: 78013755624.19.5ED3BA3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf01.hostedemail.com (Postfix) with ESMTP id 037C35001528 for ; Fri, 9 Apr 2021 19:04:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=xsjVcs0dKXDgvwkPyLRuNn8BGjcZ+u9/VPT+aQCiWxE=; b=fvCb9CS6VRSUjtEGURGh7JUAey CrlAG6UVilys94ZnzkMJ2H1/WXdLkgUYBOdo60iSMgs+cWBd5raVOQ4vbZSZTIcI/hitiCEZgz0NL YbXSGOuyhmPXuj5qRwE6TIPyzy48PosJyGYMAMMxavYtvuGGYxJFkxn7E2eoTDcGUNE1w5jP2xpff yMDXjjGSW5MwhIACG3rOaeS5Y6VhS4t3zqSZ5W0bYUgQdobAmktLiihA25DcdL+Ye6vYNdedTA+eR zMbN9nhctVMTI8zaNuFZaCMA4UDJ9B+YdFA5rTEzH+h96/2SR3qz07UOswlZNbZ0U6DOaBkpdTNuB VgcKGhNg==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwPH-000o7W-CF; Fri, 09 Apr 2021 19:03:18 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Christoph Hellwig , Jeff Layton Subject: [PATCH v7 16/28] mm/memcg: Add folio wrappers for various functions Date: Fri, 9 Apr 2021 19:50:53 +0100 Message-Id: <20210409185105.188284-17-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 037C35001528 X-Stat-Signature: 7ywaxgp4xwr1h3kwfe94bbgdkgctkjcs Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf01; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617995090-167205 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add new wrapper functions folio_memcg(), lock_folio_memcg(), unlock_folio_memcg(), mem_cgroup_folio_lruvec() and count_memcg_folio_event() Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- include/linux/memcontrol.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b8b0a802852c..f15b46f5b06c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -456,6 +456,11 @@ static inline struct mem_cgroup *page_memcg(struct page *page) return __page_memcg(page); } +static inline struct mem_cgroup *folio_memcg(struct folio *folio) +{ + return page_memcg(&folio->page); +} + /* * page_memcg_rcu - locklessly get the memory cgroup associated with a page * @page: a pointer to the page struct @@ -1052,6 +1057,15 @@ static inline void count_memcg_page_event(struct page *page, count_memcg_events(memcg, idx, 1); } +static inline void count_memcg_folio_event(struct folio *folio, + enum vm_event_item idx) +{ + struct mem_cgroup *memcg = folio_memcg(folio); + + if (memcg) + count_memcg_events(memcg, idx, folio_nr_pages(folio)); +} + static inline void count_memcg_event_mm(struct mm_struct *mm, enum vm_event_item idx) { @@ -1471,6 +1485,22 @@ unsigned long mem_cgroup_soft_limit_reclaim(pg_data_t *pgdat, int order, } #endif /* CONFIG_MEMCG */ +static inline void lock_folio_memcg(struct folio *folio) +{ + lock_page_memcg(&folio->page); +} + +static inline void unlock_folio_memcg(struct folio *folio) +{ + unlock_page_memcg(&folio->page); +} + +static inline struct lruvec *mem_cgroup_folio_lruvec(struct folio *folio, + struct pglist_data *pgdat) +{ + return mem_cgroup_page_lruvec(&folio->page, pgdat); +} + static inline void __inc_lruvec_kmem_state(void *p, enum node_stat_item idx) { __mod_lruvec_kmem_state(p, idx, 1); From patchwork Fri Apr 9 18:51:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 549CCC433B4 for ; Fri, 9 Apr 2021 19:11:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BFAC66115B for ; Fri, 9 Apr 2021 19:11:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BFAC66115B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4C58F6B006C; Fri, 9 Apr 2021 15:11:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 475E76B006E; Fri, 9 Apr 2021 15:11:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 316706B0070; Fri, 9 Apr 2021 15:11:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0111.hostedemail.com [216.40.44.111]) by kanga.kvack.org (Postfix) with ESMTP id 10F906B006C for ; Fri, 9 Apr 2021 15:11:20 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B47551F1B for ; Fri, 9 Apr 2021 19:11:19 +0000 (UTC) X-FDA: 78013771878.02.E9DB379 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf26.hostedemail.com (Postfix) with ESMTP id 3FCAF40002DA for ; Fri, 9 Apr 2021 19:11:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=c6bF0o530sEFfUOMYKiuz98B39kHRlFATNnIYOB0g+s=; b=NLy3Kc8IJA0Z9nJex8rikMcRPq kFaBIqGGoW0m2L0+vR03kv6LYIgM0CUtRY/idHgu8/gEnMXHYGhyPIAQfTvk7m4RQg6lERJZauk3c JmlkoPe6pffNiyAV6WRxA+AA1/reFta1fkrDzqqwG05sNCFL7kyNiUmA0yXEl6uABbp8Vzgw/EajQ mi1x6GEocmC78mAXCVgs4G9bjXdmXqM5lUD65wv8pEkyXzBAliZeOyzVIluQZQyIumgn538n8RpPc YVypm1imLLt+HUpsBy8pV3QFa37S9czp9BtcTVwadTFdQ7mU6Xe/gB3yF4ri7/ZojIhUhxyc7DTrB TbmRVOuw==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwVY-000oZV-25; Fri, 09 Apr 2021 19:09:48 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Christoph Hellwig , Jeff Layton Subject: [PATCH v7 25/28] mm/writeback: Add wait_for_stable_folio Date: Fri, 9 Apr 2021 19:51:02 +0100 Message-Id: <20210409185105.188284-26-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3FCAF40002DA X-Stat-Signature: o3ucgpa1uad918f1fq8gy9ksg7sse4ni Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617995476-398788 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move wait_for_stable_page() into the folio compatibility file. wait_for_stable_folio() avoids a call to compound_head() and is 14 bytes smaller than wait_for_stable_page() was. The net text size grows by 24 bytes as a result of this patch. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- include/linux/pagemap.h | 1 + mm/folio-compat.c | 6 ++++++ mm/page-writeback.c | 24 ++++++++++++++---------- 3 files changed, 21 insertions(+), 10 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 99331c35c89c..d50fc5adbee1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -833,6 +833,7 @@ int wait_on_folio_writeback_killable(struct folio *folio); void end_page_writeback(struct page *page); void end_folio_writeback(struct folio *folio); void wait_for_stable_page(struct page *page); +void wait_for_stable_folio(struct folio *folio); void page_endio(struct page *page, bool is_write, int err); diff --git a/mm/folio-compat.c b/mm/folio-compat.c index 6aadecc39fba..335594fe414e 100644 --- a/mm/folio-compat.c +++ b/mm/folio-compat.c @@ -29,3 +29,9 @@ void wait_on_page_writeback(struct page *page) return wait_on_folio_writeback(page_folio(page)); } EXPORT_SYMBOL_GPL(wait_on_page_writeback); + +void wait_for_stable_page(struct page *page) +{ + return wait_for_stable_folio(page_folio(page)); +} +EXPORT_SYMBOL_GPL(wait_for_stable_page); diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 8271f9b24b69..9d55ceec05c0 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2865,17 +2865,21 @@ int wait_on_folio_writeback_killable(struct folio *folio) EXPORT_SYMBOL_GPL(wait_on_folio_writeback_killable); /** - * wait_for_stable_page() - wait for writeback to finish, if necessary. - * @page: The page to wait on. + * wait_for_stable_folio() - wait for writeback to finish, if necessary. + * @folio: The folio to wait on. * - * This function determines if the given page is related to a backing device - * that requires page contents to be held stable during writeback. If so, then - * it will wait for any pending writeback to complete. + * This function determines if the given folio is related to a backing + * device that requires folio contents to be held stable during writeback. + * If so, then it will wait for any pending writeback to complete. + * + * Context: Sleeps. Must be called in process context and with + * no spinlocks held. Caller should hold a reference on the folio. + * If the folio is not locked, writeback may start again after writeback + * has finished. */ -void wait_for_stable_page(struct page *page) +void wait_for_stable_folio(struct folio *folio) { - page = thp_head(page); - if (page->mapping->host->i_sb->s_iflags & SB_I_STABLE_WRITES) - wait_on_page_writeback(page); + if (folio->mapping->host->i_sb->s_iflags & SB_I_STABLE_WRITES) + wait_on_folio_writeback(folio); } -EXPORT_SYMBOL_GPL(wait_for_stable_page); +EXPORT_SYMBOL_GPL(wait_for_stable_folio); From patchwork Fri Apr 9 18:51:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 12194941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56400C433B4 for ; Fri, 9 Apr 2021 19:12:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E3FD061165 for ; Fri, 9 Apr 2021 19:12:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E3FD061165 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 783A36B006C; Fri, 9 Apr 2021 15:12:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7334D6B006E; Fri, 9 Apr 2021 15:12:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FC8A6B0070; Fri, 9 Apr 2021 15:12:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id 46B1A6B006C for ; Fri, 9 Apr 2021 15:12:32 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 02C93824805A for ; Fri, 9 Apr 2021 19:12:32 +0000 (UTC) X-FDA: 78013774944.38.8CE8038 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf16.hostedemail.com (Postfix) with ESMTP id 22E9F80192C0 for ; Fri, 9 Apr 2021 19:12:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=jAqapOL6/Q7ges0ll0u4wMviUWG3R6VVJlANPuTC5Tc=; b=plkPzPgJ9L3NdtLtOy7OCld88H 4ANipG3gob05ukIQJwXhmCy8Z3UkEilYFLQYu7bEpFvhSsRu4WDbcbnU1KjPaD7/aZlqU9YMesdav D3VSL9ETfUu5mygvMfdCsEKuXCjoRKtEzNugxs6W4O+mk8uSg8BZSP3hINEp5XhQLPnS2AMADg8Cr l8EZQ5j605Qu8x7ZXXGwDMZsmHs+6rUPkUt4veW4R2QM9tAYGyky5V9zgAwlvSQUMlstFNwHMUtak REEHW+jJQOrnnBywA7jHeRK1lQGddO0lQeMrd8KaZq3sJ/j9Rwyz+t6TqMCLUI9D7ZG187gV5yHai TwzWzAWg==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lUwXB-000oiB-BF; Fri, 09 Apr 2021 19:11:22 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-cachefs@redhat.com, linux-afs@lists.infradead.org, Christoph Hellwig , Jeff Layton Subject: [PATCH v7 27/28] mm/filemap: Convert wake_up_page_bit to wake_up_folio_bit Date: Fri, 9 Apr 2021 19:51:04 +0100 Message-Id: <20210409185105.188284-28-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210409185105.188284-1-willy@infradead.org> References: <20210409185105.188284-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 22E9F80192C0 X-Stat-Signature: boy6epqqsqkecp3an3xuem365txbhoty Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617995551-658121 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: All callers have a folio, so use it directly. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig Acked-by: Jeff Layton --- mm/filemap.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 8f07e21a8f29..dfdc04130c5b 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1121,14 +1121,14 @@ static int wake_page_function(wait_queue_entry_t *wait, unsigned mode, int sync, return (flags & WQ_FLAG_EXCLUSIVE) != 0; } -static void wake_up_page_bit(struct page *page, int bit_nr) +static void wake_up_folio_bit(struct folio *folio, int bit_nr) { - wait_queue_head_t *q = page_waitqueue(page); + wait_queue_head_t *q = page_waitqueue(&folio->page); struct wait_page_key key; unsigned long flags; wait_queue_entry_t bookmark; - key.page = page; + key.page = &folio->page; key.bit_nr = bit_nr; key.page_match = 0; @@ -1163,7 +1163,7 @@ static void wake_up_page_bit(struct page *page, int bit_nr) * page waiters. */ if (!waitqueue_active(q) || !key.page_match) { - ClearPageWaiters(page); + ClearFolioWaiters(folio); /* * It's possible to miss clearing Waiters here, when we woke * our page waiters, but the hashed waitqueue has waiters for @@ -1179,7 +1179,7 @@ static void wake_up_folio(struct folio *folio, int bit) { if (!FolioWaiters(folio)) return; - wake_up_page_bit(&folio->page, bit); + wake_up_folio_bit(folio, bit); } /* @@ -1444,7 +1444,7 @@ void unlock_folio(struct folio *folio) BUILD_BUG_ON(PG_waiters != 7); VM_BUG_ON_FOLIO(!FolioLocked(folio), folio); if (clear_bit_unlock_is_negative_byte(PG_locked, folio_flags(folio, 0))) - wake_up_page_bit(&folio->page, PG_locked); + wake_up_folio_bit(folio, PG_locked); } EXPORT_SYMBOL(unlock_folio); @@ -1461,11 +1461,12 @@ EXPORT_SYMBOL(unlock_folio); */ void end_page_private_2(struct page *page) { - page = compound_head(page); - VM_BUG_ON_PAGE(!PagePrivate2(page), page); - clear_bit_unlock(PG_private_2, &page->flags); - wake_up_page_bit(page, PG_private_2); - put_page(page); + struct folio *folio = page_folio(page); + + VM_BUG_ON_FOLIO(!FolioPrivate2(folio), folio); + clear_bit_unlock(PG_private_2, folio_flags(folio, 0)); + wake_up_folio_bit(folio, PG_private_2); + put_folio(folio); } EXPORT_SYMBOL(end_page_private_2);