From patchwork Wed Jun 10 20:12:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598679 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 482DF138C for ; Wed, 10 Jun 2020 20:14:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0B5BA2074B for ; Wed, 10 Jun 2020 20:14:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="OXf4GvZc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B5BA2074B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3354C8D0012; Wed, 10 Jun 2020 16:13:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2B9A58D000E; Wed, 10 Jun 2020 16:13:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A4A28D0012; Wed, 10 Jun 2020 16:13:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0236.hostedemail.com [216.40.44.236]) by kanga.kvack.org (Postfix) with ESMTP id ED5A48D000E for ; Wed, 10 Jun 2020 16:13:53 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 9B5D5181ABEA4 for ; Wed, 10 Jun 2020 20:13:53 +0000 (UTC) X-FDA: 76914403146.14.burn23_2a041f426dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 6AFA6180B69DA for ; Wed, 10 Jun 2020 20:13:53 +0000 (UTC) X-Spam-Summary: 2,0,0,d10cd8cd5877a95a,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1540:1711:1714:1730:1747:1777:1792:2393:2559:2562:2892:3138:3139:3140:3141:3142:3350:3865:3871:5007:6261:6653:7576:8957:10004:11026:11473:11658:11914:12043:12297:12555:12895:13069:13311:13357:13894:14181:14384:14394:14721:21080:21451:21627:30054:30075,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: burn23_2a041f426dce X-Filterd-Recvd-Size: 2393 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=kKWPr83CCnHxxtwHeEEuUXJQCYa+w6gxRXIKGEN1m+Y=; b=OXf4GvZchXc6QIc+E9EWDGpoXa b0FYw3Hv3hDk7d6Y8R3NO6NMgpO3eAu39v4kfFYvvF0zioGarbKKkiBbMRD85YicAXpqlhq8cJll2 jjDhMWGd3cmCdmLScyIHKBbA4rr00ssIoR7eIwFjQodO6L9iE+SyCIWtBd1ugLkSvzh9VtaFunCnO 5CU0Y/EnCaC6pj67tcNy33cGxWgUiBwbN/V3lAmOhGhpi4b2jCh/e8xEL9rfMBpxpW1CH+eVC+G7F 3zLzer60FS3bxocfGvmaz/Vn10mhTwrWYCxQci1DbFMgnq6qc6UeGABpl2sdUEnvWjU6xaeA7f1Wp 8Z+zcqpQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003Sr-8m; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 01/51] mm: Print head flags in dump_page Date: Wed, 10 Jun 2020 13:12:55 -0700 Message-Id: <20200610201345.13273-2-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6AFA6180B69DA X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Tail page flags contain very little useful information. Print the head page's flags instead (even though PageHead is a little misleading). Signed-off-by: Matthew Wilcox (Oracle) --- mm/debug.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/debug.c b/mm/debug.c index b5b1de8c71ac..384eef80649d 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -163,7 +163,7 @@ void __dump_page(struct page *page, const char *reason) out_mapping: BUILD_BUG_ON(ARRAY_SIZE(pageflag_names) != __NR_PAGEFLAGS + 1); - pr_warn("%sflags: %#lx(%pGp)%s\n", type, page->flags, &page->flags, + pr_warn("%sflags: %#lx(%pGp)%s\n", type, head->flags, &head->flags, page_cma ? " CMA" : ""); hex_only: From patchwork Wed Jun 10 20:12:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598873 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8E37090 for ; Wed, 10 Jun 2020 20:40:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5170C206A4 for ; Wed, 10 Jun 2020 20:40:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="EPazLIDr" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5170C206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9DBD18D0038; Wed, 10 Jun 2020 16:39:42 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 967668D0033; Wed, 10 Jun 2020 16:39:42 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 76DB98D0038; Wed, 10 Jun 2020 16:39:42 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0091.hostedemail.com [216.40.44.91]) by kanga.kvack.org (Postfix) with ESMTP id 4AB158D0033 for ; Wed, 10 Jun 2020 16:39:42 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0508E801C632 for ; Wed, 10 Jun 2020 20:39:42 +0000 (UTC) X-FDA: 76914468204.16.vest39_360e5ce26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id CB876100CADAE for ; Wed, 10 Jun 2020 20:39:41 +0000 (UTC) X-Spam-Summary: 2,0,0,bc910cfb0df3c2aa,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:560:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3870:3872:3874:5007:6261:6653:7576:9163:10004:11026:11658:11914:12043:12296:12297:12555:12895:13069:13161:13229:13311:13357:13894:14181:14384:14394:14721:21080:21433:21451:21627:30003:30029:30034:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: vest39_360e5ce26dce X-Filterd-Recvd-Size: 3020 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=dF5YEmT9Y0mz257W7R1ii4B+4s7NOgT8Bu3BT5oY2ho=; b=EPazLIDr93LkbLUXA/XEkeIaqW M4D79yYSexbJQKp4e7KgytNN1RX1Nja+Sf2bh71IH+fgX9WefFpfjzq3AjwsoykssxyFGtS5KUr55 56VWS40jx1q3jaLhLRK04asv4UV4fKk2EPMm9myplwpUCa+xuu6ydF8i3d5KjPZQUktBBQH5Qy1sM srz+sg4k/ihuArvqE0GnoZj2vWQWawyqnpwH6WJGW5L1cIKz9TelPRgraZVkKNdDzFM5SFa4zFNoi wfeN4vKsMjoejlApTe2M6X5zQozWPwcroZYe1JmW0LgwLGHiV794dsCexOSIy7R+hLKPSVBrZAvR6 26HRoWrA==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003Sv-9t; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 02/51] mm: Print the inode number in dump_page Date: Wed, 10 Jun 2020 13:12:56 -0700 Message-Id: <20200610201345.13273-3-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: CB876100CADAE X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" The inode number helps correlate this page with debug messages elsewhere in the kernel. Signed-off-by: Matthew Wilcox (Oracle) --- mm/debug.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/debug.c b/mm/debug.c index 384eef80649d..e30e35b41d0e 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -133,15 +133,16 @@ void __dump_page(struct page *page, const char *reason) goto out_mapping; } - if (probe_kernel_read(&dentry_first, - &host->i_dentry.first, sizeof(struct hlist_node *))) { + if (probe_kernel_read(&dentry_first, &host->i_dentry.first, + sizeof(struct hlist_node *))) { pr_warn("mapping->a_ops:%ps with invalid mapping->host inode address %px\n", a_ops, host); goto out_mapping; } if (!dentry_first) { - pr_warn("mapping->a_ops:%ps\n", a_ops); + pr_warn("mapping->a_ops:%ps ino %lx\n", a_ops, + host->i_ino); goto out_mapping; } @@ -156,8 +157,8 @@ void __dump_page(struct page *page, const char *reason) * crash, but it's unlikely that we reach here with a * corrupted struct page */ - pr_warn("mapping->aops:%ps dentry name:\"%pd\"\n", - a_ops, &dentry); + pr_warn("mapping->aops:%ps ino %lx dentry name:\"%pd\"\n", + a_ops, host->i_ino, &dentry); } } out_mapping: From patchwork Wed Jun 10 20:12:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598877 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DAEDC90 for ; Wed, 10 Jun 2020 20:40:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A78D9206A4 for ; Wed, 10 Jun 2020 20:40:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="dE4Xknev" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A78D9206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E96718D0040; Wed, 10 Jun 2020 16:39:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E6C068D0033; Wed, 10 Jun 2020 16:39:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D341B8D0040; Wed, 10 Jun 2020 16:39:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0175.hostedemail.com [216.40.44.175]) by kanga.kvack.org (Postfix) with ESMTP id BD3538D0033 for ; Wed, 10 Jun 2020 16:39:44 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 77B6F181ABEA4 for ; Wed, 10 Jun 2020 20:39:44 +0000 (UTC) X-FDA: 76914468288.11.moon36_1d16fd226dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 4B3C9180FA281 for ; Wed, 10 Jun 2020 20:39:44 +0000 (UTC) X-Spam-Summary: 2,0,0,712dfbfa085f1d26,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3352:3865:3866:4321:5007:6261:6653:7576:8957:10004:11026:11658:11914:12043:12297:12438:12555:12895:13069:13161:13229:13311:13357:13894:14181:14384:14394:14721:21080:21433:21451:21627:30003:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: moon36_1d16fd226dce X-Filterd-Recvd-Size: 3250 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=DV4ftIEdQNHLrdsOYQ9aZc9MP7MJ7x8fQ7A37yrCky8=; b=dE4XknevV4TlrEoLcQIxjzKRp+ BibugNzNqwmBXjnp0W2fFr9fGISsZu07NDmp0kSLEqS83i+m2dUsKWpQw/pFlTb1pxMyCr4uVuIZJ GCk49SOlcF2PJhRvYT6T+KR9MifKirfYnoFeDLeSfElIwmWAxGwzNiqBo9c2O1+oSllVoaVE7wnRs 0uYwfG/lPWO+08wl3MxXZLN0FVYDZUoHAiSw8qY2f/gx9f6r1qTbzXncCQ2Txxjugr8/e3cNjRoRx EKJBRl6912H2mNkP5vOjk6SYY4VqoVShGASvdRKYvS7NJEOu6V885vJx11f5spixepoqqPfv3Frp3 03+2HSxg==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003T0-At; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 03/51] mm: Print hashed address of struct page Date: Wed, 10 Jun 2020 13:12:57 -0700 Message-Id: <20200610201345.13273-4-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4B3C9180FA281 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" The actual address of the struct page isn't particularly helpful, while the hashed address helps match with other messages elsewhere. Signed-off-by: Matthew Wilcox (Oracle) --- mm/debug.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/debug.c b/mm/debug.c index e30e35b41d0e..b17909c16a77 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -86,23 +86,23 @@ void __dump_page(struct page *page, const char *reason) if (compound) if (hpage_pincount_available(page)) { - pr_warn("page:%px refcount:%d mapcount:%d mapping:%p " - "index:%#lx head:%px order:%u " + pr_warn("page:%p refcount:%d mapcount:%d mapping:%p " + "index:%#lx head:%p order:%u " "compound_mapcount:%d compound_pincount:%d\n", page, page_ref_count(head), mapcount, mapping, page_to_pgoff(page), head, compound_order(head), compound_mapcount(page), compound_pincount(page)); } else { - pr_warn("page:%px refcount:%d mapcount:%d mapping:%p " - "index:%#lx head:%px order:%u " + pr_warn("page:%p refcount:%d mapcount:%d mapping:%p " + "index:%#lx head:%p order:%u " "compound_mapcount:%d\n", page, page_ref_count(head), mapcount, mapping, page_to_pgoff(page), head, compound_order(head), compound_mapcount(page)); } else - pr_warn("page:%px refcount:%d mapcount:%d mapping:%p index:%#lx\n", + pr_warn("page:%p refcount:%d mapcount:%d mapping:%p index:%#lx\n", page, page_ref_count(page), mapcount, mapping, page_to_pgoff(page)); if (PageKsm(page)) From patchwork Wed Jun 10 20:12:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598647 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B1956138C for ; Wed, 10 Jun 2020 20:13:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 731102074B for ; Wed, 10 Jun 2020 20:13:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="bAfUEC9x" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 731102074B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 774538D0008; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6FC818D0002; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CDAE8D0007; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0179.hostedemail.com [216.40.44.179]) by kanga.kvack.org (Postfix) with ESMTP id 416BB8D0002 for ; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id F35998013423 for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) X-FDA: 76914402936.21.boats52_441679826dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id BB425180442ED for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) X-Spam-Summary: 2,0,0,19f22ce7104e8384,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2901:3138:3139:3140:3141:3142:3352:3865:3866:3867:3870:3871:3874:4385:5007:6261:6653:7576:10004:11026:11232:11638:11658:11914:12296:12297:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14721:21063:21080:21451:21627:21740:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:49,LUA_SUMMARY:none X-HE-Tag: boats52_441679826dce X-Filterd-Recvd-Size: 2610 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=VHXm1L+rObZHMg/ZFaxYy0B7In8iJsSQq6tf5Ix1Zow=; b=bAfUEC9x/4a1ubraNWUhB2jMUA wgZpuusnFmtFWjFCNm5HIyObzZT+pnK8D4l1cleAR398txg/NM5yrWcqn7zHxUMnHxTDA7PDfOuDw 03yZP0DPHgA/tCkMQFX1oAF20OYZAUGhALaBS/9OOR13Cts8cxPylP1T6u5SqJSlrBE0zk94MxbOU OPTGJX+bUeD05te31Hyqsxicqc+YRQZh/n2YdCvWQ4VP2RsdqLQTiC7O7SppH/0rXws9UsQNOFZdH e1rOsqC03rOmUAthexl+Pyv6ghP0UHyhKN8n4YZfsc664UKSCfQSP41zPU7PbpLV1dFj3VjAQ1mf1 RSJ6GprA==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003T4-C0; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 04/51] mm: Move PageDoubleMap bit Date: Wed, 10 Jun 2020 13:12:58 -0700 Message-Id: <20200610201345.13273-5-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: BB425180442ED X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" PG_private_2 is defined as being PF_ANY (applicable to tail pages as well as regular & head pages). That means that the first tail page of a double-map page will appear to have Private2 set. Use the Workingset bit instead which is defined as PF_HEAD so any attempt to access the Workingset bit on a tail page will redirect to the head page's Workingset bit. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan --- include/linux/page-flags.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 222f6f7b2bb3..de6e0696f55c 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -164,7 +164,7 @@ enum pageflags { PG_slob_free = PG_private, /* Compound pages. Stored in first tail page's flags */ - PG_double_map = PG_private_2, + PG_double_map = PG_workingset, /* non-lru isolated movable page */ PG_isolated = PG_reclaim, From patchwork Wed Jun 10 20:12:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598649 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1300814B7 for ; Wed, 10 Jun 2020 20:13:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C890220842 for ; Wed, 10 Jun 2020 20:13:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="OHI7Zx84" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C890220842 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C7DFD8D000A; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BB7188D0007; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 770B28D0007; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0004.hostedemail.com [216.40.44.4]) by kanga.kvack.org (Postfix) with ESMTP id 580B48D0006 for ; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0FBB71A1E0E for ; Wed, 10 Jun 2020 20:13:49 +0000 (UTC) X-FDA: 76914402978.15.table06_5d12b4d26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id D6C3818014EB4 for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) X-Spam-Summary: 2,0,0,f226bc01f100fdc9,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:69:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2393:2559:2562:2910:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3871:4321:5007:6261:6653:7576:8603:8957:9592:10004:11026:11232:11473:11638:11658:11914:12043:12296:12297:12438:12555:12683:12895:12986:13894:14110:14181:14394:14721:21080:21451:21627:21990:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: table06_5d12b4d26dce X-Filterd-Recvd-Size: 4542 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=8tGdVMSWeas8R2bSKE6AqR3+R5LRIfaVDMdjwwPPyK0=; b=OHI7Zx84YXLGrqDuDmcU2fBDIQ Kce38e89HwRoAR6r6yYkS316prKjI/zonLSocwIMFA8JXimr/Jx3HITAO5qwxyt0Zk1gf8vs7ZDw9 sxp1p1xyYCpCTItaIQyURY+x/aBoSbu5046/kdbaE5mFzs7n7S1YUCpnG8q64AxnvIJww0hGNSovM De9VvULcZNijF16zF1/4c6QMKy/aEPY1xUrunMFRes4flE6ylxT/BbN/AcLesZAnDKc0PXZNiKJpN 4gvAiJv5ont6iJ7JvYb/RJrk3kL13CO/dAONGdw4Sp7Vg19qQQNILBldWj9EGKpl0aJF4Y/aI2s7/ 4H7ejRIw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003TF-DG; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 05/51] mm: Simplify PageDoubleMap with PF_SECOND policy Date: Wed, 10 Jun 2020 13:12:59 -0700 Message-Id: <20200610201345.13273-6-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D6C3818014EB4 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Introduce the new page policy of PF_SECOND which lets us use the normal pageflags generation machinery to create the various DoubleMap manipulation functions. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/page-flags.h | 40 ++++++++++---------------------------- 1 file changed, 10 insertions(+), 30 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index de6e0696f55c..979460df4768 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -232,6 +232,9 @@ static inline void page_init_poison(struct page *page, size_t size) * * PF_NO_COMPOUND: * the page flag is not relevant for compound pages. + * + * PF_SECOND: + * the page flag is stored in the first tail page. */ #define PF_POISONED_CHECK(page) ({ \ VM_BUG_ON_PGFLAGS(PagePoisoned(page), page); \ @@ -247,6 +250,9 @@ static inline void page_init_poison(struct page *page, size_t size) #define PF_NO_COMPOUND(page, enforce) ({ \ VM_BUG_ON_PGFLAGS(enforce && PageCompound(page), page); \ PF_POISONED_CHECK(page); }) +#define PF_SECOND(page, enforce) ({ \ + VM_BUG_ON_PGFLAGS(!PageHead(page), page); \ + PF_POISONED_CHECK(&page[1]); }) /* * Macros to create function definitions for page flags @@ -685,42 +691,15 @@ static inline int PageTransTail(struct page *page) * * See also __split_huge_pmd_locked() and page_remove_anon_compound_rmap(). */ -static inline int PageDoubleMap(struct page *page) -{ - return PageHead(page) && test_bit(PG_double_map, &page[1].flags); -} - -static inline void SetPageDoubleMap(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHead(page), page); - set_bit(PG_double_map, &page[1].flags); -} - -static inline void ClearPageDoubleMap(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHead(page), page); - clear_bit(PG_double_map, &page[1].flags); -} -static inline int TestSetPageDoubleMap(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHead(page), page); - return test_and_set_bit(PG_double_map, &page[1].flags); -} - -static inline int TestClearPageDoubleMap(struct page *page) -{ - VM_BUG_ON_PAGE(!PageHead(page), page); - return test_and_clear_bit(PG_double_map, &page[1].flags); -} - +PAGEFLAG(DoubleMap, double_map, PF_SECOND) + TESTSCFLAG(DoubleMap, double_map, PF_SECOND) #else TESTPAGEFLAG_FALSE(TransHuge) TESTPAGEFLAG_FALSE(TransCompound) TESTPAGEFLAG_FALSE(TransCompoundMap) TESTPAGEFLAG_FALSE(TransTail) PAGEFLAG_FALSE(DoubleMap) - TESTSETFLAG_FALSE(DoubleMap) - TESTCLEARFLAG_FALSE(DoubleMap) + TESTSCFLAG_FALSE(DoubleMap) #endif /* @@ -875,6 +854,7 @@ static inline int page_has_private(struct page *page) #undef PF_ONLY_HEAD #undef PF_NO_TAIL #undef PF_NO_COMPOUND +#undef PF_SECOND #endif /* !__GENERATING_BOUNDS_H */ #endif /* PAGE_FLAGS_H */ From patchwork Wed Jun 10 20:13:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598673 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A9E45138C for ; Wed, 10 Jun 2020 20:14:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 76731207F9 for ; Wed, 10 Jun 2020 20:14:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="LZhl8Xoq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 76731207F9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C13818D000F; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AAA978D0011; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74EB58D000C; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0166.hostedemail.com [216.40.44.166]) by kanga.kvack.org (Postfix) with ESMTP id 4E8D98D000E for ; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 07734181ABEA4 for ; Wed, 10 Jun 2020 20:13:52 +0000 (UTC) X-FDA: 76914403104.01.beds66_200658226dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id B8A551802E626 for ; Wed, 10 Jun 2020 20:13:51 +0000 (UTC) X-Spam-Summary: 2,0,0,57dffe6822a62b96,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:560:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3352:3865:3866:3867:3870:3871:4321:4605:5007:6261:6653:7576:8957:10004:11026:11473:11658:11914:12043:12296:12297:12438:12555:12895:12986:13894:14096:14181:14394:14721:21080:21451:21627:21990:30034:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: beds66_200658226dce X-Filterd-Recvd-Size: 4141 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=J/UeCwTdxdvssMS23HfBywrddunudah3jTHxFdDluWw=; b=LZhl8XoqWmoxoe4oiI7dUinI5V U776oT0ROF8uRC9JNB/LJXhTM+4p3TiyRvjzsIpCYCyO3a68xtitko/SBbg+ZVmx5GhngKvg9teOe WdyM9DdbiWNiNitspHxiYl9Nnk76ZbMXbemycCOEyXCj0YGes58HTegT39DkLVzMaCjRFXStIJ5kG CCvcIZMcZ+8zG/h8Ltb42ivV8fr2o3mwB1mfvj4o/3/wV0RG+WfDKZzyxnfPYIYRbsnTLZiVYQP3i bDBLGubh523Oi32hOAU2iXSukuT2B54nfqXae++cWaFptGmzzUymDbzAPIFUGkPFbaJp3uSHJUErK qj72UUNw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003TO-EQ; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 06/51] mm: Store compound_nr as well as compound_order Date: Wed, 10 Jun 2020 13:13:00 -0700 Message-Id: <20200610201345.13273-7-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: B8A551802E626 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" This removes a few instructions from functions which need to know how many pages are in a compound page. The storage used is either page->mapping on 64-bit or page->index on 32-bit. Both of these are fine to overlay on tail pages. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 5 ++++- include/linux/mm_types.h | 1 + mm/page_alloc.c | 5 +++-- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index dc7b87310c10..af0305ad090f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -911,12 +911,15 @@ static inline int compound_pincount(struct page *page) static inline void set_compound_order(struct page *page, unsigned int order) { page[1].compound_order = order; + page[1].compound_nr = 1U << order; } /* Returns the number of pages in this potentially compound page. */ static inline unsigned long compound_nr(struct page *page) { - return 1UL << compound_order(page); + if (!PageHead(page)) + return 1; + return page[1].compound_nr; } /* Returns the number of bytes in this potentially compound page. */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 64ede5f150dc..561ed987ab44 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -134,6 +134,7 @@ struct page { unsigned char compound_dtor; unsigned char compound_order; atomic_t compound_mapcount; + unsigned int compound_nr; /* 1 << compound_order */ }; struct { /* Second tail page of compound page */ unsigned long _compound_pad_1; /* compound_head */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 727751219003..3fb61ef4c3a4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -673,8 +673,6 @@ void prep_compound_page(struct page *page, unsigned int order) int i; int nr_pages = 1 << order; - set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); - set_compound_order(page, order); __SetPageHead(page); for (i = 1; i < nr_pages; i++) { struct page *p = page + i; @@ -682,6 +680,9 @@ void prep_compound_page(struct page *page, unsigned int order) p->mapping = TAIL_MAPPING; set_compound_head(p, page); } + + set_compound_page_dtor(page, COMPOUND_PAGE_DTOR); + set_compound_order(page, order); atomic_set(compound_mapcount_ptr(page), -1); if (hpage_pincount_available(page)) atomic_set(compound_pincount_ptr(page), 0); From patchwork Wed Jun 10 20:13:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598653 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF689138C for ; Wed, 10 Jun 2020 20:13:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 718542074B for ; Wed, 10 Jun 2020 20:13:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ES+Sfxy+" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 718542074B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0952D8D0002; Wed, 10 Jun 2020 16:13:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C2E618D0009; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E1C68D000A; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0036.hostedemail.com [216.40.44.36]) by kanga.kvack.org (Postfix) with ESMTP id 7A4038D0009 for ; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 3D9D018030326 for ; Wed, 10 Jun 2020 20:13:49 +0000 (UTC) X-FDA: 76914402978.06.meat48_4e06e6026dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 00D3F10222869 for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) X-Spam-Summary: 2,0,0,d9ee059430346470,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3868:3870:3872:3874:5007:6261:6653:7576:7903:8957:10004:11026:11658:11914:12043:12297:12438:12555:12895:13069:13161:13229:13311:13357:13894:14096:14181:14384:14394:14721:21080:21451:21627:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: meat48_4e06e6026dce X-Filterd-Recvd-Size: 2811 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=a6bntLUvQ0tR7N/aMeU2zpyWKO2+RHf9NYJ0TB23leA=; b=ES+Sfxy+2oGIgIAmOW/I+p76fw qcPQqsG/73jemyJRer3+UjVnFFYLHVhkXkhsOZxvbDVdfaxrs8dIjC8cBBcdzK7zb/1YAMGtViobv E4OSMFg5GQlh+bhKZZ51XuWVi+onf+ucWA1IeqIffN5derpB0OCxfCNSGwMrj0wvaB68R571Ft4b0 jS/v42UxtraBVF1FSe9nchpywSSnisn5Gu9+L5VfjTS/jm1k9qBVx01aejUktPjYOXQE77kQhM8Ml 8VKqo4AarBlMKAm7s/S+867z77I9uroThzBOvWBQHlR+1afGTF+QxbeIt6kkmu5ycnhjB0nhxY/W1 sf0IMA9w==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003TU-Gq; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 07/51] mm: Move page-flags include to top of file Date: Wed, 10 Jun 2020 13:13:01 -0700 Message-Id: <20200610201345.13273-8-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 00D3F10222869 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Give up on the notion that we can remove page-flags.h from mm.h. There are currently 14 inline functions which use a PageFoo function. Also, two of the files directly included by mm.h include page-flags.h themselves, and there are probably more indirect inclusions. So just include it at the top like any other header file. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/mm.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index af0305ad090f..6c29b663135f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -667,11 +668,6 @@ int vma_is_stack_for_current(struct vm_area_struct *vma); struct mmu_gather; struct inode; -/* - * FIXME: take this include out, include page-flags.h in - * files which need it (119 of them) - */ -#include #include /* From patchwork Wed Jun 10 20:13:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598659 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E571714B7 for ; Wed, 10 Jun 2020 20:14:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B177A2074B for ; Wed, 10 Jun 2020 20:14:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="eQkWOqFz" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B177A2074B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 907B48D000B; Wed, 10 Jun 2020 16:13:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 57C718D0009; Wed, 10 Jun 2020 16:13:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 21E398D0007; Wed, 10 Jun 2020 16:13:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id DDF6D8D0006 for ; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9A3751A1E1C for ; Wed, 10 Jun 2020 20:13:49 +0000 (UTC) X-FDA: 76914402978.14.net56_580603726dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin14.hostedemail.com (Postfix) with ESMTP id 5AB7E181016E7 for ; Wed, 10 Jun 2020 20:13:49 +0000 (UTC) X-Spam-Summary: 2,0,0,10b410fd77bd4e74,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:560:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3352:3865:3866:3867:3871:3872:4321:5007:6261:6653:7576:7875:8957:10004:11026:11473:11658:11914:12043:12296:12297:12438:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21451:21627:21966:21990:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: net56_580603726dce X-Filterd-Recvd-Size: 2993 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=rN/mlFQb7HaxIMdRoCBWWTWQlk5OHj7v1RP+jX7Yp4Y=; b=eQkWOqFzpT9FwkkpBW+oRka0dP MuhOIi4m2vJrWWRLdz1++H4mVbdMhV527yj1MrHCeXjSD+s10sC1Hx8RzZEgfAZ3K4ezxW6zsDPNz PUp7ckg/fBq13+BALbTfR87DIzU3yGwHp/59R30N6jpiQsKBWWRIQ2NNbtxF2X3L0FH4bJwA1UYnu GiXKKAI6+15lvMolQwyh7uP6GmN3ts6bwQbHVdpiA1yqE/AfC1l4Iw0jySUprEsHSpgKoVbAqUfuF LO1ynifb8BGqeso/F5sa35PicEPeraxP+ToPUlr5sE67ev85/ncOJuO6fLYmz3Vb1Y/RT9uthemeE OHuocJ2Q==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003Tf-I1; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 08/51] mm: Add thp_order Date: Wed, 10 Jun 2020 13:13:02 -0700 Message-Id: <20200610201345.13273-9-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5AB7E181016E7 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" This function returns the order of a transparent huge page. It compiles to 0 if CONFIG_TRANSPARENT_HUGEPAGE is disabled. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/huge_mm.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 71f20776b06c..dd19720a8bc2 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -265,6 +265,19 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud, else return NULL; } + +/** + * thp_order - Order of a transparent huge page. + * @page: Head page of a transparent huge page. + */ +static inline unsigned int thp_order(struct page *page) +{ + VM_BUG_ON_PGFLAGS(PageTail(page), page); + if (PageHead(page)) + return HPAGE_PMD_ORDER; + return 0; +} + static inline int hpage_nr_pages(struct page *page) { if (unlikely(PageTransHuge(page))) @@ -324,6 +337,12 @@ static inline struct list_head *page_deferred_list(struct page *page) #define HPAGE_PUD_MASK ({ BUILD_BUG(); 0; }) #define HPAGE_PUD_SIZE ({ BUILD_BUG(); 0; }) +static inline unsigned int thp_order(struct page *page) +{ + VM_BUG_ON_PGFLAGS(PageTail(page), page); + return 0; +} + static inline int hpage_nr_pages(struct page *page) { VM_BUG_ON_PAGE(PageTail(page), page); From patchwork Wed Jun 10 20:13:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598867 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFB60618 for ; Wed, 10 Jun 2020 20:40:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B2C83206A4 for ; Wed, 10 Jun 2020 20:40:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="QwUVdJWy" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B2C83206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1EE368D0035; Wed, 10 Jun 2020 16:39:37 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 178578D0033; Wed, 10 Jun 2020 16:39:37 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E96CC8D0035; Wed, 10 Jun 2020 16:39:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0052.hostedemail.com [216.40.44.52]) by kanga.kvack.org (Postfix) with ESMTP id C52FE8D0033 for ; Wed, 10 Jun 2020 16:39:36 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6A04816926 for ; Wed, 10 Jun 2020 20:39:36 +0000 (UTC) X-FDA: 76914467952.21.lace05_0b04bd926dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id 36B5F18043C62 for ; Wed, 10 Jun 2020 20:39:36 +0000 (UTC) X-Spam-Summary: 2,0,0,e317797852510f2e,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:371:372:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3871:3872:4321:4605:5007:6119:6261:6653:7576:7875:7903:8603:10004:11026:11473:11658:11914:12296:12297:12438:12555:12895:12986:13894:14096:14181:14394:14721:21080:21451:21627:21966:21990:30003:30034:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: lace05_0b04bd926dce X-Filterd-Recvd-Size: 5826 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=hN/37aRhgzzO2Cu+X3jM7dH/y9JI16FETpuj+qe6qfM=; b=QwUVdJWyV9AHqtytLyaTzSg1RZ kKswsI3XPSy5+PmHAHPn+IveskrPZkD0lYCK/fdORj5YqbXlf8G6s9Dof2YbI9MRAN3tsIrsZ96hG E0lGakXh9umHZL1X9hL/6QIHgX7aSvFrNX1RHgeLvolIXInjJ6egbVKpwLEFNH63bsxx509TgCDSW ZaLRdHZgqSQWM9FR2vOjSaENRdjvsgKbDYKoBNjlrM+yEufRk1DOREwTa365Q4gtnmVDNgpltmZUz uxeHZCGRVQQrkch+HQL0YIX8PCkPYLG5c2mwiIqeed6eILT99YYWw96fyldiEh4vXfC/g956oIZkE Llbi9A2A==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003Tn-J6; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 09/51] mm: Add thp_size Date: Wed, 10 Jun 2020 13:13:03 -0700 Message-Id: <20200610201345.13273-10-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 36B5F18043C62 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" This function returns the number of bytes in a THP. It is like page_size(), but compiles to just PAGE_SIZE if CONFIG_TRANSPARENT_HUGEPAGE is disabled. Signed-off-by: Matthew Wilcox (Oracle) --- drivers/nvdimm/btt.c | 4 +--- drivers/nvdimm/pmem.c | 6 ++---- include/linux/huge_mm.h | 11 +++++++++++ mm/internal.h | 2 +- mm/page_io.c | 2 +- mm/page_vma_mapped.c | 4 ++-- 6 files changed, 18 insertions(+), 11 deletions(-) diff --git a/drivers/nvdimm/btt.c b/drivers/nvdimm/btt.c index 90c0c4bbe77b..7255cfe6ebe2 100644 --- a/drivers/nvdimm/btt.c +++ b/drivers/nvdimm/btt.c @@ -1490,10 +1490,8 @@ static int btt_rw_page(struct block_device *bdev, sector_t sector, { struct btt *btt = bdev->bd_disk->private_data; int rc; - unsigned int len; - len = hpage_nr_pages(page) * PAGE_SIZE; - rc = btt_do_bvec(btt, NULL, page, len, 0, op, sector); + rc = btt_do_bvec(btt, NULL, page, thp_size(page), 0, op, sector); if (rc == 0) page_endio(page, op_is_write(op), 0); diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c index d1ecd6da11a2..d1999c266b20 100644 --- a/drivers/nvdimm/pmem.c +++ b/drivers/nvdimm/pmem.c @@ -238,11 +238,9 @@ static int pmem_rw_page(struct block_device *bdev, sector_t sector, blk_status_t rc; if (op_is_write(op)) - rc = pmem_do_write(pmem, page, 0, sector, - hpage_nr_pages(page) * PAGE_SIZE); + rc = pmem_do_write(pmem, page, 0, sector, thp_size(page)); else - rc = pmem_do_read(pmem, page, 0, sector, - hpage_nr_pages(page) * PAGE_SIZE); + rc = pmem_do_read(pmem, page, 0, sector, thp_size(page)); /* * The ->rw_page interface is subtle and tricky. The core * retries on any error, so we can only invoke page_endio() in diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index dd19720a8bc2..0ec3b5a73d38 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -469,4 +469,15 @@ static inline bool thp_migration_supported(void) } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +/** + * thp_size - Size of a transparent huge page. + * @page: Head page of a transparent huge page. + * + * Return: Number of bytes in this page. + */ +static inline unsigned long thp_size(struct page *page) +{ + return PAGE_SIZE << thp_order(page); +} + #endif /* _LINUX_HUGE_MM_H */ diff --git a/mm/internal.h b/mm/internal.h index 9886db20d94f..de9f1d0ba5fc 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -395,7 +395,7 @@ vma_address(struct page *page, struct vm_area_struct *vma) unsigned long start, end; start = __vma_address(page, vma); - end = start + PAGE_SIZE * (hpage_nr_pages(page) - 1); + end = start + thp_size(page) - PAGE_SIZE; /* page should be within @vma mapping range */ VM_BUG_ON_VMA(end < vma->vm_start || start >= vma->vm_end, vma); diff --git a/mm/page_io.c b/mm/page_io.c index e8726f3e3820..888000d1a8cc 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -40,7 +40,7 @@ static struct bio *get_swap_bio(gfp_t gfp_flags, bio->bi_iter.bi_sector <<= PAGE_SHIFT - 9; bio->bi_end_io = end_io; - bio_add_page(bio, page, PAGE_SIZE * hpage_nr_pages(page), 0); + bio_add_page(bio, page, thp_size(page), 0); } return bio; } diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 719c35246cfa..e65629c056e8 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -227,7 +227,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) if (pvmw->address >= pvmw->vma->vm_end || pvmw->address >= __vma_address(pvmw->page, pvmw->vma) + - hpage_nr_pages(pvmw->page) * PAGE_SIZE) + thp_size(pvmw->page)) return not_found(pvmw); /* Did we cross page table boundary? */ if (pvmw->address % PMD_SIZE == 0) { @@ -268,7 +268,7 @@ int page_mapped_in_vma(struct page *page, struct vm_area_struct *vma) unsigned long start, end; start = __vma_address(page, vma); - end = start + PAGE_SIZE * (hpage_nr_pages(page) - 1); + end = start + thp_size(page) - PAGE_SIZE; if (unlikely(end < vma->vm_start || start >= vma->vm_end)) return 0; From patchwork Wed Jun 10 20:13:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598875 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3E6CA90 for ; Wed, 10 Jun 2020 20:40:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D82C9206A4 for ; Wed, 10 Jun 2020 20:40:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="aZWDacIo" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D82C9206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 412DC8D003B; Wed, 10 Jun 2020 16:39:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 349718D0033; Wed, 10 Jun 2020 16:39:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1ED9C8D003B; Wed, 10 Jun 2020 16:39:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0042.hostedemail.com [216.40.44.42]) by kanga.kvack.org (Postfix) with ESMTP id D53588D0033 for ; Wed, 10 Jun 2020 16:39:43 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 8A54A13F8E9 for ; Wed, 10 Jun 2020 20:39:43 +0000 (UTC) X-FDA: 76914468246.25.money46_3a098e126dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 33D4E1804E3E1 for ; Wed, 10 Jun 2020 20:39:43 +0000 (UTC) X-Spam-Summary: 2,0,0,5a09e4ca2adfcd62,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:327:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2553:2559:2562:2693:2898:2904:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:4321:4385:4605:5007:6119:6261:6653:7576:7875:7903:8603:8957:10004:11026:11232:11233:11473:11658:11914:12043:12296:12297:12438:12555:12895:12986:13161:13229:13894:14394:21080:21324:21433:21450:21451:21611:21627:21966:21990:30003:30034:30054:30070:30079:30090,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: money46_3a098e126dce X-Filterd-Recvd-Size: 24330 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=S+KYvksaZek96grFu9+YDx/5aDvWfDI7UeetEIGhohQ=; b=aZWDacIo0vgkjFA6WkSZvHUDsF SUaG1d6mPOPwL39l42W21WGVojMiDO9C1WvKjXUxGXvSsl0Nxcuo98JAx5eOx8+f6Xpc8l37mKTJB TCzj5PrddSMqBjwT4QEZjJuTHk+VC3jqY+zbDPKSe7bLo2vTmdwEZ2GpCTjoUe/rRW2Rt0wrxJsNw QpQ4mH4oiqnQMkIFZgNiAGqE4vrJppPWugHuJjTgFXbC5TNqPBUttuhdleG0ADhNfXqytJFDPCOwb 9o1im0qHII9MWqALYc+THgQSfSq86cIZkb/r/21CqwhCtF9fiPHc581Pk77/FH2LTe4xIFiV+PoLm 38uKQNDw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003Tr-KS; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 10/51] mm: Replace hpage_nr_pages with thp_nr_pages Date: Wed, 10 Jun 2020 13:13:04 -0700 Message-Id: <20200610201345.13273-11-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 33D4E1804E3E1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" The thp prefix is more frequently used than hpage and we should be consistent between the various functions. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/huge_mm.h | 13 +++++++++---- include/linux/mm_inline.h | 6 +++--- include/linux/pagemap.h | 6 +++--- mm/compaction.c | 2 +- mm/filemap.c | 2 +- mm/gup.c | 2 +- mm/hugetlb.c | 2 +- mm/internal.h | 2 +- mm/memcontrol.c | 10 +++++----- mm/memory_hotplug.c | 7 +++---- mm/mempolicy.c | 2 +- mm/migrate.c | 16 ++++++++-------- mm/mlock.c | 9 ++++----- mm/page_io.c | 2 +- mm/page_vma_mapped.c | 2 +- mm/rmap.c | 8 ++++---- mm/swap.c | 16 ++++++++-------- mm/swap_state.c | 6 +++--- mm/swapfile.c | 2 +- mm/vmscan.c | 6 +++--- 20 files changed, 62 insertions(+), 59 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 0ec3b5a73d38..dcdfd21763a3 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -278,9 +278,14 @@ static inline unsigned int thp_order(struct page *page) return 0; } -static inline int hpage_nr_pages(struct page *page) +/** + * thp_nr_pages - The number of regular pages in this huge page. + * @page: The head page of a huge page. + */ +static inline int thp_nr_pages(struct page *page) { - if (unlikely(PageTransHuge(page))) + VM_BUG_ON_PGFLAGS(PageTail(page), page); + if (PageHead(page)) return HPAGE_PMD_NR; return 1; } @@ -343,9 +348,9 @@ static inline unsigned int thp_order(struct page *page) return 0; } -static inline int hpage_nr_pages(struct page *page) +static inline int thp_nr_pages(struct page *page) { - VM_BUG_ON_PAGE(PageTail(page), page); + VM_BUG_ON_PGFLAGS(PageTail(page), page); return 1; } diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 219bef41d87c..8fc71e9d7bb0 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -48,14 +48,14 @@ static __always_inline void update_lru_size(struct lruvec *lruvec, static __always_inline void add_page_to_lru_list(struct page *page, struct lruvec *lruvec, enum lru_list lru) { - update_lru_size(lruvec, lru, page_zonenum(page), hpage_nr_pages(page)); + update_lru_size(lruvec, lru, page_zonenum(page), thp_nr_pages(page)); list_add(&page->lru, &lruvec->lists[lru]); } static __always_inline void add_page_to_lru_list_tail(struct page *page, struct lruvec *lruvec, enum lru_list lru) { - update_lru_size(lruvec, lru, page_zonenum(page), hpage_nr_pages(page)); + update_lru_size(lruvec, lru, page_zonenum(page), thp_nr_pages(page)); list_add_tail(&page->lru, &lruvec->lists[lru]); } @@ -63,7 +63,7 @@ static __always_inline void del_page_from_lru_list(struct page *page, struct lruvec *lruvec, enum lru_list lru) { list_del(&page->lru); - update_lru_size(lruvec, lru, page_zonenum(page), -hpage_nr_pages(page)); + update_lru_size(lruvec, lru, page_zonenum(page), -thp_nr_pages(page)); } /** diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index cf2468da68e9..484a36185bb5 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -381,7 +381,7 @@ static inline struct page *find_subpage(struct page *head, pgoff_t index) if (PageHuge(head)) return head; - return head + (index & (hpage_nr_pages(head) - 1)); + return head + (index & (thp_nr_pages(head) - 1)); } struct page *find_get_entry(struct address_space *mapping, pgoff_t offset); @@ -730,7 +730,7 @@ static inline struct page *readahead_page(struct readahead_control *rac) page = xa_load(&rac->mapping->i_pages, rac->_index); VM_BUG_ON_PAGE(!PageLocked(page), page); - rac->_batch_count = hpage_nr_pages(page); + rac->_batch_count = thp_nr_pages(page); return page; } @@ -753,7 +753,7 @@ static inline unsigned int __readahead_batch(struct readahead_control *rac, VM_BUG_ON_PAGE(!PageLocked(page), page); VM_BUG_ON_PAGE(PageTail(page), page); array[i++] = page; - rac->_batch_count += hpage_nr_pages(page); + rac->_batch_count += thp_nr_pages(page); /* * The page cache isn't using multi-index entries yet, diff --git a/mm/compaction.c b/mm/compaction.c index fd988b7e5f2b..81eaffcfbe4e 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -991,7 +991,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, del_page_from_lru_list(page, lruvec, page_lru(page)); mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + page_is_file_lru(page), - hpage_nr_pages(page)); + thp_nr_pages(page)); isolate_success: list_add(&page->lru, &cc->migratepages); diff --git a/mm/filemap.c b/mm/filemap.c index f0ae9a6308cb..80ce3658b147 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -197,7 +197,7 @@ static void unaccount_page_cache_page(struct address_space *mapping, if (PageHuge(page)) return; - nr = hpage_nr_pages(page); + nr = thp_nr_pages(page); __mod_lruvec_page_state(page, NR_FILE_PAGES, -nr); if (PageSwapBacked(page)) { diff --git a/mm/gup.c b/mm/gup.c index de9e36262ccb..12d066baee11 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1703,7 +1703,7 @@ static long check_and_migrate_cma_pages(struct task_struct *tsk, mod_node_page_state(page_pgdat(head), NR_ISOLATED_ANON + page_is_file_lru(head), - hpage_nr_pages(head)); + thp_nr_pages(head)); } } } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 57ece74e3aae..6bb07bc655f7 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1593,7 +1593,7 @@ static struct address_space *_get_hugetlb_page_mapping(struct page *hpage) /* Use first found vma */ pgoff_start = page_to_pgoff(hpage); - pgoff_end = pgoff_start + hpage_nr_pages(hpage) - 1; + pgoff_end = pgoff_start + thp_nr_pages(hpage) - 1; anon_vma_interval_tree_foreach(avc, &anon_vma->rb_root, pgoff_start, pgoff_end) { struct vm_area_struct *vma = avc->vma; diff --git a/mm/internal.h b/mm/internal.h index de9f1d0ba5fc..ac3c79408045 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -368,7 +368,7 @@ extern void clear_page_mlock(struct page *page); static inline void mlock_migrate_page(struct page *newpage, struct page *page) { if (TestClearPageMlocked(page)) { - int nr_pages = hpage_nr_pages(page); + int nr_pages = thp_nr_pages(page); /* Holding pmd lock, no change in irq context: __mod is safe */ __mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0b38b6ad547d..af02455db4b3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5363,7 +5363,7 @@ static int mem_cgroup_move_account(struct page *page, { struct lruvec *from_vec, *to_vec; struct pglist_data *pgdat; - unsigned int nr_pages = compound ? hpage_nr_pages(page) : 1; + unsigned int nr_pages = compound ? thp_nr_pages(page) : 1; int ret; VM_BUG_ON(from == to); @@ -6453,7 +6453,7 @@ enum mem_cgroup_protection mem_cgroup_protected(struct mem_cgroup *root, */ int mem_cgroup_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask) { - unsigned int nr_pages = hpage_nr_pages(page); + unsigned int nr_pages = thp_nr_pages(page); struct mem_cgroup *memcg = NULL; int ret = 0; @@ -6684,7 +6684,7 @@ void mem_cgroup_migrate(struct page *oldpage, struct page *newpage) return; /* Force-charge the new page. The old one will be freed soon */ - nr_pages = hpage_nr_pages(newpage); + nr_pages = thp_nr_pages(newpage); page_counter_charge(&memcg->memory, nr_pages); if (do_memsw_account()) @@ -6897,7 +6897,7 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry) * ancestor for the swap instead and transfer the memory+swap charge. */ swap_memcg = mem_cgroup_id_get_online(memcg); - nr_entries = hpage_nr_pages(page); + nr_entries = thp_nr_pages(page); /* Get references for the tail pages, too */ if (nr_entries > 1) mem_cgroup_id_get_many(swap_memcg, nr_entries - 1); @@ -6942,7 +6942,7 @@ void mem_cgroup_swapout(struct page *page, swp_entry_t entry) */ int mem_cgroup_try_charge_swap(struct page *page, swp_entry_t entry) { - unsigned int nr_pages = hpage_nr_pages(page); + unsigned int nr_pages = thp_nr_pages(page); struct page_counter *counter; struct mem_cgroup *memcg; unsigned short oldid; diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index c4d5c45820d0..dfbff3565dad 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1253,7 +1253,7 @@ static int do_migrate_range(unsigned long start_pfn, unsigned long end_pfn) { unsigned long pfn; - struct page *page; + struct page *page, *head; int ret = 0; LIST_HEAD(source); @@ -1261,15 +1261,14 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn) if (!pfn_valid(pfn)) continue; page = pfn_to_page(pfn); + head = compound_head(page); if (PageHuge(page)) { - struct page *head = compound_head(page); pfn = page_to_pfn(head) + compound_nr(head) - 1; isolate_huge_page(head, &source); continue; } else if (PageTransHuge(page)) - pfn = page_to_pfn(compound_head(page)) - + hpage_nr_pages(page) - 1; + pfn = page_to_pfn(head) + thp_nr_pages(page) - 1; /* * HWPoison pages have elevated reference counts so the migration would diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 381320671677..d2b11c291e78 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1049,7 +1049,7 @@ static int migrate_page_add(struct page *page, struct list_head *pagelist, list_add_tail(&head->lru, pagelist); mod_node_page_state(page_pgdat(head), NR_ISOLATED_ANON + page_is_file_lru(head), - hpage_nr_pages(head)); + thp_nr_pages(head)); } else if (flags & MPOL_MF_STRICT) { /* * Non-movable page may reach here. And, there may be diff --git a/mm/migrate.c b/mm/migrate.c index f37729673558..9d0c6a853c1c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -193,7 +193,7 @@ void putback_movable_pages(struct list_head *l) put_page(page); } else { mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + - page_is_file_lru(page), -hpage_nr_pages(page)); + page_is_file_lru(page), -thp_nr_pages(page)); putback_lru_page(page); } } @@ -386,7 +386,7 @@ static int expected_page_refs(struct address_space *mapping, struct page *page) */ expected_count += is_device_private_page(page); if (mapping) - expected_count += hpage_nr_pages(page) + page_has_private(page); + expected_count += thp_nr_pages(page) + page_has_private(page); return expected_count; } @@ -441,7 +441,7 @@ int migrate_page_move_mapping(struct address_space *mapping, */ newpage->index = page->index; newpage->mapping = page->mapping; - page_ref_add(newpage, hpage_nr_pages(page)); /* add cache reference */ + page_ref_add(newpage, thp_nr_pages(page)); /* add cache reference */ if (PageSwapBacked(page)) { __SetPageSwapBacked(newpage); if (PageSwapCache(page)) { @@ -474,7 +474,7 @@ int migrate_page_move_mapping(struct address_space *mapping, * to one less reference. * We know this isn't the last reference. */ - page_ref_unfreeze(page, expected_count - hpage_nr_pages(page)); + page_ref_unfreeze(page, expected_count - thp_nr_pages(page)); xas_unlock(&xas); /* Leave irq disabled to prevent preemption while updating stats */ @@ -591,7 +591,7 @@ static void copy_huge_page(struct page *dst, struct page *src) } else { /* thp page */ BUG_ON(!PageTransHuge(src)); - nr_pages = hpage_nr_pages(src); + nr_pages = thp_nr_pages(src); } for (i = 0; i < nr_pages; i++) { @@ -1224,7 +1224,7 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page, */ if (likely(!__PageMovable(page))) mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + - page_is_file_lru(page), -hpage_nr_pages(page)); + page_is_file_lru(page), -thp_nr_pages(page)); } /* @@ -1598,7 +1598,7 @@ static int add_page_for_migration(struct mm_struct *mm, unsigned long addr, list_add_tail(&head->lru, pagelist); mod_node_page_state(page_pgdat(head), NR_ISOLATED_ANON + page_is_file_lru(head), - hpage_nr_pages(head)); + thp_nr_pages(head)); } out_putpage: /* @@ -1962,7 +1962,7 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) page_lru = page_is_file_lru(page); mod_node_page_state(page_pgdat(page), NR_ISOLATED_ANON + page_lru, - hpage_nr_pages(page)); + thp_nr_pages(page)); /* * Isolating the page has taken another reference, so the diff --git a/mm/mlock.c b/mm/mlock.c index f8736136fad7..93ca2bf30b4f 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -61,8 +61,7 @@ void clear_page_mlock(struct page *page) if (!TestClearPageMlocked(page)) return; - mod_zone_page_state(page_zone(page), NR_MLOCK, - -hpage_nr_pages(page)); + mod_zone_page_state(page_zone(page), NR_MLOCK, -thp_nr_pages(page)); count_vm_event(UNEVICTABLE_PGCLEARED); /* * The previous TestClearPageMlocked() corresponds to the smp_mb() @@ -95,7 +94,7 @@ void mlock_vma_page(struct page *page) if (!TestSetPageMlocked(page)) { mod_zone_page_state(page_zone(page), NR_MLOCK, - hpage_nr_pages(page)); + thp_nr_pages(page)); count_vm_event(UNEVICTABLE_PGMLOCKED); if (!isolate_lru_page(page)) putback_lru_page(page); @@ -192,7 +191,7 @@ unsigned int munlock_vma_page(struct page *page) /* * Serialize with any parallel __split_huge_page_refcount() which * might otherwise copy PageMlocked to part of the tail pages before - * we clear it in the head page. It also stabilizes hpage_nr_pages(). + * we clear it in the head page. It also stabilizes thp_nr_pages(). */ spin_lock_irq(&pgdat->lru_lock); @@ -202,7 +201,7 @@ unsigned int munlock_vma_page(struct page *page) goto unlock_out; } - nr_pages = hpage_nr_pages(page); + nr_pages = thp_nr_pages(page); __mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages); if (__munlock_isolate_lru_page(page, true)) { diff --git a/mm/page_io.c b/mm/page_io.c index 888000d1a8cc..77170b7e6f04 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -274,7 +274,7 @@ static inline void count_swpout_vm_event(struct page *page) if (unlikely(PageTransHuge(page))) count_vm_event(THP_SWPOUT); #endif - count_vm_events(PSWPOUT, hpage_nr_pages(page)); + count_vm_events(PSWPOUT, thp_nr_pages(page)); } int __swap_writepage(struct page *page, struct writeback_control *wbc, diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index e65629c056e8..5e77b269c330 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -61,7 +61,7 @@ static inline bool pfn_is_match(struct page *page, unsigned long pfn) return page_pfn == pfn; /* THP can be referenced by any subpage */ - return pfn >= page_pfn && pfn - page_pfn < hpage_nr_pages(page); + return pfn >= page_pfn && pfn - page_pfn < thp_nr_pages(page); } /** diff --git a/mm/rmap.c b/mm/rmap.c index 5fe2dedce1fc..c56fab5826c1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1130,7 +1130,7 @@ void do_page_add_anon_rmap(struct page *page, } if (first) { - int nr = compound ? hpage_nr_pages(page) : 1; + int nr = compound ? thp_nr_pages(page) : 1; /* * We use the irq-unsafe __{inc|mod}_zone_page_stat because * these counters are not modified in interrupt context, and @@ -1169,7 +1169,7 @@ void do_page_add_anon_rmap(struct page *page, void page_add_new_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address, bool compound) { - int nr = compound ? hpage_nr_pages(page) : 1; + int nr = compound ? thp_nr_pages(page) : 1; VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma); __SetPageSwapBacked(page); @@ -1860,7 +1860,7 @@ static void rmap_walk_anon(struct page *page, struct rmap_walk_control *rwc, return; pgoff_start = page_to_pgoff(page); - pgoff_end = pgoff_start + hpage_nr_pages(page) - 1; + pgoff_end = pgoff_start + thp_nr_pages(page) - 1; anon_vma_interval_tree_foreach(avc, &anon_vma->rb_root, pgoff_start, pgoff_end) { struct vm_area_struct *vma = avc->vma; @@ -1913,7 +1913,7 @@ static void rmap_walk_file(struct page *page, struct rmap_walk_control *rwc, return; pgoff_start = page_to_pgoff(page); - pgoff_end = pgoff_start + hpage_nr_pages(page) - 1; + pgoff_end = pgoff_start + thp_nr_pages(page) - 1; if (!locked) i_mmap_lock_read(mapping); vma_interval_tree_foreach(vma, &mapping->i_mmap, diff --git a/mm/swap.c b/mm/swap.c index dbcab84c6fce..a3b8cc8bdc17 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -241,7 +241,7 @@ static void pagevec_move_tail_fn(struct page *page, struct lruvec *lruvec, del_page_from_lru_list(page, lruvec, page_lru(page)); ClearPageActive(page); add_page_to_lru_list_tail(page, lruvec, page_lru(page)); - (*pgmoved) += hpage_nr_pages(page); + (*pgmoved) += thp_nr_pages(page); } } @@ -312,7 +312,7 @@ void lru_note_cost(struct lruvec *lruvec, bool file, unsigned int nr_pages) void lru_note_cost_page(struct page *page) { lru_note_cost(mem_cgroup_page_lruvec(page, page_pgdat(page)), - page_is_file_lru(page), hpage_nr_pages(page)); + page_is_file_lru(page), thp_nr_pages(page)); } static void __activate_page(struct page *page, struct lruvec *lruvec, @@ -320,7 +320,7 @@ static void __activate_page(struct page *page, struct lruvec *lruvec, { if (PageLRU(page) && !PageActive(page) && !PageUnevictable(page)) { int lru = page_lru_base_type(page); - int nr_pages = hpage_nr_pages(page); + int nr_pages = thp_nr_pages(page); del_page_from_lru_list(page, lruvec, lru); SetPageActive(page); @@ -500,7 +500,7 @@ void lru_cache_add_active_or_unevictable(struct page *page, * lock is held(spinlock), which implies preemption disabled. */ __mod_zone_page_state(page_zone(page), NR_MLOCK, - hpage_nr_pages(page)); + thp_nr_pages(page)); count_vm_event(UNEVICTABLE_PGMLOCKED); } lru_cache_add(page); @@ -532,7 +532,7 @@ static void lru_deactivate_file_fn(struct page *page, struct lruvec *lruvec, { int lru; bool active; - int nr_pages = hpage_nr_pages(page); + int nr_pages = thp_nr_pages(page); if (!PageLRU(page)) return; @@ -580,7 +580,7 @@ static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec, { if (PageLRU(page) && PageActive(page) && !PageUnevictable(page)) { int lru = page_lru_base_type(page); - int nr_pages = hpage_nr_pages(page); + int nr_pages = thp_nr_pages(page); del_page_from_lru_list(page, lruvec, lru + LRU_ACTIVE); ClearPageActive(page); @@ -599,7 +599,7 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec, if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) && !PageSwapCache(page) && !PageUnevictable(page)) { bool active = PageActive(page); - int nr_pages = hpage_nr_pages(page); + int nr_pages = thp_nr_pages(page); del_page_from_lru_list(page, lruvec, LRU_INACTIVE_ANON + active); @@ -972,7 +972,7 @@ static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec, { enum lru_list lru; int was_unevictable = TestClearPageUnevictable(page); - int nr_pages = hpage_nr_pages(page); + int nr_pages = thp_nr_pages(page); VM_BUG_ON_PAGE(PageLRU(page), page); diff --git a/mm/swap_state.c b/mm/swap_state.c index e98ff460e9e9..108d2574371f 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -115,7 +115,7 @@ int add_to_swap_cache(struct page *page, swp_entry_t entry, gfp_t gfp) struct address_space *address_space = swap_address_space(entry); pgoff_t idx = swp_offset(entry); XA_STATE_ORDER(xas, &address_space->i_pages, idx, compound_order(page)); - unsigned long i, nr = hpage_nr_pages(page); + unsigned long i, nr = thp_nr_pages(page); VM_BUG_ON_PAGE(!PageLocked(page), page); VM_BUG_ON_PAGE(PageSwapCache(page), page); @@ -157,7 +157,7 @@ int add_to_swap_cache(struct page *page, swp_entry_t entry, gfp_t gfp) void __delete_from_swap_cache(struct page *page, swp_entry_t entry) { struct address_space *address_space = swap_address_space(entry); - int i, nr = hpage_nr_pages(page); + int i, nr = thp_nr_pages(page); pgoff_t idx = swp_offset(entry); XA_STATE(xas, &address_space->i_pages, idx); @@ -250,7 +250,7 @@ void delete_from_swap_cache(struct page *page) xa_unlock_irq(&address_space->i_pages); put_swap_page(page, entry); - page_ref_sub(page, hpage_nr_pages(page)); + page_ref_sub(page, thp_nr_pages(page)); } /* diff --git a/mm/swapfile.c b/mm/swapfile.c index 987276c557d1..142095774e55 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1368,7 +1368,7 @@ void put_swap_page(struct page *page, swp_entry_t entry) unsigned char *map; unsigned int i, free_entries = 0; unsigned char val; - int size = swap_entry_size(hpage_nr_pages(page)); + int size = swap_entry_size(thp_nr_pages(page)); si = _swap_info_get(entry); if (!si) diff --git a/mm/vmscan.c b/mm/vmscan.c index b6d84326bdf2..17934e03b3aa 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1359,7 +1359,7 @@ static unsigned int shrink_page_list(struct list_head *page_list, case PAGE_ACTIVATE: goto activate_locked; case PAGE_SUCCESS: - stat->nr_pageout += hpage_nr_pages(page); + stat->nr_pageout += thp_nr_pages(page); if (PageWriteback(page)) goto keep; @@ -1867,7 +1867,7 @@ static unsigned noinline_for_stack move_pages_to_lru(struct lruvec *lruvec, SetPageLRU(page); lru = page_lru(page); - nr_pages = hpage_nr_pages(page); + nr_pages = thp_nr_pages(page); update_lru_size(lruvec, lru, page_zonenum(page), nr_pages); list_move(&page->lru, &lruvec->lists[lru]); @@ -2067,7 +2067,7 @@ static void shrink_active_list(unsigned long nr_to_scan, * so we ignore them here. */ if ((vm_flags & VM_EXEC) && page_is_file_lru(page)) { - nr_rotated += hpage_nr_pages(page); + nr_rotated += thp_nr_pages(page); list_add(&page->lru, &l_active); continue; } From patchwork Wed Jun 10 20:13:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598671 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A0B9514E3 for ; Wed, 10 Jun 2020 20:14:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6C94320853 for ; Wed, 10 Jun 2020 20:14:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="enb+c39t" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6C94320853 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8B0568D0010; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 834578D000E; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 616FB8D000F; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id 3F69A8D000C for ; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E6EBB20BC9 for ; Wed, 10 Jun 2020 20:13:51 +0000 (UTC) X-FDA: 76914403062.01.horse25_62059c726dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id A95DA18022BFE for ; Wed, 10 Jun 2020 20:13:51 +0000 (UTC) X-Spam-Summary: 2,0,0,100f5a85707e19d4,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3352:3865:3866:3870:3871:4321:5007:6261:6653:7576:7875:8957:10004:11026:11473:11658:11914:12043:12296:12297:12438:12555:12895:12986:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21451:21627:21990:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: horse25_62059c726dce X-Filterd-Recvd-Size: 2920 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=gx5dzaclcqLvRwdwg62Y129njOqoREHSy9PwwIfrnGA=; b=enb+c39tttjAMEhbGFU3C4fuYh R4EasE+JPjlKSH8Sn5bzwqozpn3biKPMl0hwkAsw8XFfjkSvWEPrS7SGBFQkNs07fk5mky5CE7+qs 9qopi0rY9KJI4hB2REfzZ1iGt+uyXlr+b22RynpFdk0VNyr4jPkQj8jiJOlFBdnkmCudVD/sZMdSo bKpPoO+C5QElTvYW4bnOjLwaCf49QqhXdaJKY49MhLsKJ+JDfD4i0MtB0da5Vy92NwzQhkXnWA+HB SU4LPG1hv/DZhXQ4nOaYvdpTFPvzHmgO/bUCKV4WrAUd59s2kF1wn6m5UPJjSbzUePQDu460FbbHq i4LHWOAw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003U1-Lf; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 11/51] mm: Add thp_head Date: Wed, 10 Jun 2020 13:13:05 -0700 Message-Id: <20200610201345.13273-12-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: A95DA18022BFE X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" This is like compound_head() but compiles away when CONFIG_TRANSPARENT_HUGEPAGE is not enabled. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/huge_mm.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index dcdfd21763a3..bd13e9ac3437 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -266,6 +266,15 @@ static inline spinlock_t *pud_trans_huge_lock(pud_t *pud, return NULL; } +/** + * thp_head - Head page of a transparent huge page. + * @page: Any page (tail, head or regular) found in the page cache. + */ +static inline struct page *thp_head(struct page *page) +{ + return compound_head(page); +} + /** * thp_order - Order of a transparent huge page. * @page: Head page of a transparent huge page. @@ -342,6 +351,12 @@ static inline struct list_head *page_deferred_list(struct page *page) #define HPAGE_PUD_MASK ({ BUILD_BUG(); 0; }) #define HPAGE_PUD_SIZE ({ BUILD_BUG(); 0; }) +static inline struct page *thp_head(struct page *page) +{ + VM_BUG_ON_PGFLAGS(PageTail(page), page); + return page; +} + static inline unsigned int thp_order(struct page *page) { VM_BUG_ON_PGFLAGS(PageTail(page), page); From patchwork Wed Jun 10 20:13:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598871 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2D57390 for ; Wed, 10 Jun 2020 20:40:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EEAE9206A4 for ; Wed, 10 Jun 2020 20:40:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="kzXLN9R5" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EEAE9206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 742978D0037; Wed, 10 Jun 2020 16:39:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6A6618D0033; Wed, 10 Jun 2020 16:39:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5476B8D0037; Wed, 10 Jun 2020 16:39:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0043.hostedemail.com [216.40.44.43]) by kanga.kvack.org (Postfix) with ESMTP id 2CFBC8D0033 for ; Wed, 10 Jun 2020 16:39:41 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E52FE181ABEA4 for ; Wed, 10 Jun 2020 20:39:40 +0000 (UTC) X-FDA: 76914468120.20.smile94_3f0ff3d26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id C15EA181E6364 for ; Wed, 10 Jun 2020 20:39:40 +0000 (UTC) X-Spam-Summary: 2,0,0,7f6628b9944551fd,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1540:1711:1714:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3351:3865:3867:3868:3870:3871:3872:3873:4321:4385:5007:6261:6653:7576:8603:10004:11026:11658:11914:12043:12297:12438:12555:12895:12986:13069:13311:13357:13894:14181:14384:14394:14721:21080:21451:21627:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: smile94_3f0ff3d26dce X-Filterd-Recvd-Size: 2567 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=ZdmHI6xA8It9U2TlgFLx4kPuqY4+rF2vVFre5EFi3Ak=; b=kzXLN9R5EqQ0rOFb3IvKNQFB95 gcRS4OD44bdayszrjbTk5VR2un4Ss2fUTtW86fPimwIsWyHg9S8wTmoag/6dtNpsyaBfBRgVnuZEv VVevEOiXoa1RCDIuimQ7iOalqNeUusacHTopUoDke0oCtn/XDQkk9zDB4d4bWZDnEWxyQDV/PqNlo dZGOScNRnJpoyKJWYoSeakEOz2CQ03uOWGE+ZXJE+MW3iGv/pI2KqfA1DAbQQn5E3s7PjtXvFMqSl OvH/poDMCVaIQB/DZa3mOYICYxSbiH+e3zsvgr43516uZdgqTE+ZZMRAUWXkOFKPC/Qhn7KYnvg/T E2AE87Ng==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003U6-NE; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, David Hildenbrand Subject: [PATCH v6 12/51] mm: Introduce offset_in_thp Date: Wed, 10 Jun 2020 13:13:06 -0700 Message-Id: <20200610201345.13273-13-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C15EA181E6364 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Mirroring offset_in_page(), this gives you the offset within this particular page, no matter what size page it is. It optimises down to offset_in_page() if CONFIG_TRANSPARENT_HUGEPAGE is not set. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: David Hildenbrand --- include/linux/mm.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 6c29b663135f..3fc7e8121216 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1583,6 +1583,7 @@ static inline void clear_page_pfmemalloc(struct page *page) extern void pagefault_out_of_memory(void); #define offset_in_page(p) ((unsigned long)(p) & ~PAGE_MASK) +#define offset_in_thp(page, p) ((unsigned long)(p) & (thp_size(page) - 1)) /* * Flags passed to show_mem() and show_free_areas() to suppress output in From patchwork Wed Jun 10 20:13:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598675 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A3B4D14B7 for ; Wed, 10 Jun 2020 20:14:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 646D62074B for ; Wed, 10 Jun 2020 20:14:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="aQpqFIA5" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 646D62074B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 069A48D000C; Wed, 10 Jun 2020 16:13:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F158B8D0013; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0F0B8D000C; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id 85AC78D000F for ; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 4946C801869A for ; Wed, 10 Jun 2020 20:13:52 +0000 (UTC) X-FDA: 76914403104.17.sofa13_040f18126dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 0F74218024DFE for ; Wed, 10 Jun 2020 20:13:52 +0000 (UTC) X-Spam-Summary: 2,0,0,db28bcc028a1d6aa,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1543:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3354:3865:3866:3870:4250:4321:4605:5007:6119:6120:6261:6653:7576:7901:8603:8957:9592:10004:11026:11473:11658:11914:12043:12291:12297:12438:12555:12683:12895:12986:13894:14110:14181:14394:14721:21080:21451:21627:21990:30034:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:0,LUA_SUMMARY:none X-HE-Tag: sofa13_040f18126dce X-Filterd-Recvd-Size: 4603 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=To8irju2/LINnhXcN1nryanQ2IhjjlZJW8yhv6eYHp0=; b=aQpqFIA5a558mRuswO/tjXDNEn CHb/HxwhlS9KyjDOKbpVv1YyD5U3BJFb0R/HHmy4xtop6G73Q+metDN8dyXElNUyr/i3IP5XFjOsu 6vk2uqzdXcdPm+kEQPcb5ZWvc6AbfXXc6WoXf+te+SKLdkMNgpalGnvRWoGy8fpHp3dkZtVJ5ZKHR +qB5ebtCOIx/ATPsPtQatEc0ORAWmedZC6KVzqk3F08IFBY7Y7EX8xFu8AucLmeA+crs6Nt6dKP/e X7Xm1jeENjj3I0v44tyH7wXU2Gi+tZ+88MDYbxQ5RX/aQx5olU91P22O9yGZVq2wPrgKpXRdH0GkK FFlk/wqA==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003UF-OB; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 13/51] mm: Support arbitrary THP sizes Date: Wed, 10 Jun 2020 13:13:07 -0700 Message-Id: <20200610201345.13273-14-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0F74218024DFE X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Use the compound size of the page instead of assuming PTE or PMD size. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/huge_mm.h | 8 ++------ include/linux/mm.h | 42 ++++++++++++++++++++--------------------- 2 files changed, 23 insertions(+), 27 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index bd13e9ac3437..d125912a3e0d 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -282,9 +282,7 @@ static inline struct page *thp_head(struct page *page) static inline unsigned int thp_order(struct page *page) { VM_BUG_ON_PGFLAGS(PageTail(page), page); - if (PageHead(page)) - return HPAGE_PMD_ORDER; - return 0; + return compound_order(page); } /** @@ -294,9 +292,7 @@ static inline unsigned int thp_order(struct page *page) static inline int thp_nr_pages(struct page *page) { VM_BUG_ON_PGFLAGS(PageTail(page), page); - if (PageHead(page)) - return HPAGE_PMD_NR; - return 1; + return compound_nr(page); } struct page *follow_devmap_pmd(struct vm_area_struct *vma, unsigned long addr, diff --git a/include/linux/mm.h b/include/linux/mm.h index 3fc7e8121216..67b36b141ec7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -668,6 +668,27 @@ int vma_is_stack_for_current(struct vm_area_struct *vma); struct mmu_gather; struct inode; +static inline unsigned int compound_order(struct page *page) +{ + if (!PageHead(page)) + return 0; + return page[1].compound_order; +} + +/* Returns the number of pages in this potentially compound page. */ +static inline unsigned long compound_nr(struct page *page) +{ + if (!PageHead(page)) + return 1; + return page[1].compound_nr; +} + +static inline void set_compound_order(struct page *page, unsigned int order) +{ + page[1].compound_order = order; + page[1].compound_nr = 1U << order; +} + #include /* @@ -879,13 +900,6 @@ static inline void destroy_compound_page(struct page *page) compound_page_dtors[page[1].compound_dtor](page); } -static inline unsigned int compound_order(struct page *page) -{ - if (!PageHead(page)) - return 0; - return page[1].compound_order; -} - static inline bool hpage_pincount_available(struct page *page) { /* @@ -904,20 +918,6 @@ static inline int compound_pincount(struct page *page) return atomic_read(compound_pincount_ptr(page)); } -static inline void set_compound_order(struct page *page, unsigned int order) -{ - page[1].compound_order = order; - page[1].compound_nr = 1U << order; -} - -/* Returns the number of pages in this potentially compound page. */ -static inline unsigned long compound_nr(struct page *page) -{ - if (!PageHead(page)) - return 1; - return page[1].compound_nr; -} - /* Returns the number of bytes in this potentially compound page. */ static inline unsigned long page_size(struct page *page) { From patchwork Wed Jun 10 20:13:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598655 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CEEE0138C for ; Wed, 10 Jun 2020 20:13:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9BDE62074B for ; Wed, 10 Jun 2020 20:13:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="LkO1X8I4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9BDE62074B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3650E8D0006; Wed, 10 Jun 2020 16:13:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D66898D000B; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B19F88D0002; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0037.hostedemail.com [216.40.44.37]) by kanga.kvack.org (Postfix) with ESMTP id 781808D0006 for ; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2171A1A1E12 for ; Wed, 10 Jun 2020 20:13:49 +0000 (UTC) X-FDA: 76914402978.02.army25_16058c726dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id F04031A1E0D for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) X-Spam-Summary: 2,0,0,18109efb4097971a,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2693:3138:3139:3140:3141:3142:3353:3865:3867:3870:4321:4605:5007:6117:6238:6261:6653:7576:7903:10004:11026:11473:11658:11914:12043:12296:12297:12438:12555:12895:13161:13229:13894:14096:14181:14394:14721:21080:21433:21451:21627:21987:21990:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: army25_16058c726dce X-Filterd-Recvd-Size: 4187 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=PUB7gfbS6ya/Oo2I8iRDMBuQXcmwMzHEvRek9lLjtrk=; b=LkO1X8I4f9ucjNuiIzDwWeYSWv tcZiSpsQMP6v0AfubRtN5yEuKdqJfX+vAeUQkz6WQ3IEqXIuuPtOHqkaudY9BnFJnxy8HhlafpghG 5gJcWMvLnrf84utweVaf3uut8m6rt9uhCww3CZPUMMBONYuglZ83iCxeOWKrVOouWFTP20af0MAGj DWS6kbuW89rGv93C9UpLVDw4yPjF2cF8BtvP38cQiBJhJ/NktrV8Bw4oyjy4xwvR7OKwMVLfSH4sJ 2qc+GO3Qp3GelzcO3dBjUrnBWZScZfR1TxX21ZmLZHJQ4WXZDw0ucQR0j+xSd3yC9quB1AZVD4854 EXhPakMQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003UJ-PQ; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 14/51] fs: Add a filesystem flag for THPs Date: Wed, 10 Jun 2020 13:13:08 -0700 Message-Id: <20200610201345.13273-15-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: F04031A1E0D X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" The page cache needs to know whether the filesystem supports THPs. Signed-off-by: Matthew Wilcox (Oracle) --- fs/inode.c | 2 ++ include/linux/fs.h | 1 + include/linux/pagemap.h | 6 ++++++ mm/shmem.c | 2 +- 4 files changed, 10 insertions(+), 1 deletion(-) diff --git a/fs/inode.c b/fs/inode.c index 72c4c347afb7..9d78c37b00b8 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -181,6 +181,8 @@ int inode_init_always(struct super_block *sb, struct inode *inode) mapping->a_ops = &empty_aops; mapping->host = inode; mapping->flags = 0; + if (sb->s_type->fs_flags & FS_THP_SUPPORT) + __set_bit(AS_THP_SUPPORT, &mapping->flags); mapping->wb_err = 0; atomic_set(&mapping->i_mmap_writable, 0); #ifdef CONFIG_READ_ONLY_THP_FOR_FS diff --git a/include/linux/fs.h b/include/linux/fs.h index 19ef6c88c152..ac411a08f83c 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2241,6 +2241,7 @@ struct file_system_type { #define FS_HAS_SUBTYPE 4 #define FS_USERNS_MOUNT 8 /* Can be mounted by userns root */ #define FS_DISALLOW_NOTIFY_PERM 16 /* Disable fanotify permission events */ +#define FS_THP_SUPPORT 8192 /* Remove once all fs converted */ #define FS_RENAME_DOES_D_MOVE 32768 /* FS will handle d_move() during rename() internally. */ int (*init_fs_context)(struct fs_context *); const struct fs_parameter_spec *parameters; diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 484a36185bb5..8a45b572bcf1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -29,6 +29,7 @@ enum mapping_flags { AS_EXITING = 4, /* final truncate in progress */ /* writeback related tags are not used */ AS_NO_WRITEBACK_TAGS = 5, + AS_THP_SUPPORT = 6, /* THPs supported */ }; /** @@ -119,6 +120,11 @@ static inline void mapping_set_gfp_mask(struct address_space *m, gfp_t mask) m->gfp_mask = mask; } +static inline bool mapping_thp_support(struct address_space *mapping) +{ + return test_bit(AS_THP_SUPPORT, &mapping->flags); +} + void release_pages(struct page **pages, int nr); /* diff --git a/mm/shmem.c b/mm/shmem.c index a0dbe62f8042..a05d129a45e9 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3854,7 +3854,7 @@ static struct file_system_type shmem_fs_type = { .parameters = shmem_fs_parameters, #endif .kill_sb = kill_litter_super, - .fs_flags = FS_USERNS_MOUNT, + .fs_flags = FS_USERNS_MOUNT | FS_THP_SUPPORT, }; int __init shmem_init(void) From patchwork Wed Jun 10 20:13:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598657 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF5F9138C for ; Wed, 10 Jun 2020 20:13:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A1CE52074B for ; Wed, 10 Jun 2020 20:13:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="t387Iyw8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A1CE52074B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 61E6A8D0007; Wed, 10 Jun 2020 16:13:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 01E868D000C; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDB338D0002; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id B40048D0006 for ; Wed, 10 Jun 2020 16:13:49 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5E0851A1E17 for ; Wed, 10 Jun 2020 20:13:49 +0000 (UTC) X-FDA: 76914402978.09.eye72_321141026dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 161BF180ABE4B for ; Wed, 10 Jun 2020 20:13:49 +0000 (UTC) X-Spam-Summary: 2,0,0,1c15f27a0f965d70,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:69:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2393:2559:2562:2692:2693:3138:3139:3140:3141:3142:3353:3865:3867:3870:3871:3872:4321:4605:5007:6117:6261:6653:7576:9592:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12555:12895:12986:13161:13184:13229:13894:14096:14181:14394:14721:21080:21451:21627:21987:21990:30054:30070,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: eye72_321141026dce X-Filterd-Recvd-Size: 4268 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=S5644RWSmzLKClPw4/cvpdrJ3oC3Vx5GXky4qEg0/Sw=; b=t387Iyw8XBmw/PCNwgPA8fTWvX FDCT5sqoJc6buh0Ea9yw6B2Mwdne1kqZXmXFEiSPqzIShANQmCPWpqt8kJbRyg6u1hd57zpjigSLB Bo2UINMBdXnjnmijOgZeH4p5NAJgF/j2o0QZDy7v7ELE1rVt7PJR8xLH2BJlaIVxyK3QlN1h+a/Dd 1G+Ivtl/Khg2942dOxVmkbayq7S0bA59/afBuDV5ELg3XHIOwlzCJHJbHnKOsCkZAFLS7ejdVfWBz HbrUUIkv2/0kRFQFAn/aryEP/3L9xA4lNXei1WUa035YzAg4Fhhs09+aWElbFTkrguFtEvMCUlpmW c/yW5gJQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003UO-Qp; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 15/51] fs: Do not update nr_thps for mappings which support THPs Date: Wed, 10 Jun 2020 13:13:09 -0700 Message-Id: <20200610201345.13273-16-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 161BF180ABE4B X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" The nr_thps counter is to support THPs in the page cache when the filesystem doesn't understand THPs. Eventually it will be removed, but we should still support filesystems which do not understand THPs yet. Move the nr_thp manipulation functions to filemap.h since they're page-cache specific. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/fs.h | 27 --------------------------- include/linux/pagemap.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 29 insertions(+), 27 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index ac411a08f83c..e11929152103 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2839,33 +2839,6 @@ static inline errseq_t file_sample_sb_err(struct file *file) return errseq_sample(&file->f_path.dentry->d_sb->s_wb_err); } -static inline int filemap_nr_thps(struct address_space *mapping) -{ -#ifdef CONFIG_READ_ONLY_THP_FOR_FS - return atomic_read(&mapping->nr_thps); -#else - return 0; -#endif -} - -static inline void filemap_nr_thps_inc(struct address_space *mapping) -{ -#ifdef CONFIG_READ_ONLY_THP_FOR_FS - atomic_inc(&mapping->nr_thps); -#else - WARN_ON_ONCE(1); -#endif -} - -static inline void filemap_nr_thps_dec(struct address_space *mapping) -{ -#ifdef CONFIG_READ_ONLY_THP_FOR_FS - atomic_dec(&mapping->nr_thps); -#else - WARN_ON_ONCE(1); -#endif -} - extern int vfs_fsync_range(struct file *file, loff_t start, loff_t end, int datasync); extern int vfs_fsync(struct file *file, int datasync); diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 8a45b572bcf1..644caff3e78b 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -125,6 +125,35 @@ static inline bool mapping_thp_support(struct address_space *mapping) return test_bit(AS_THP_SUPPORT, &mapping->flags); } +static inline int filemap_nr_thps(struct address_space *mapping) +{ +#ifdef CONFIG_READ_ONLY_THP_FOR_FS + return atomic_read(&mapping->nr_thps); +#else + return 0; +#endif +} + +static inline void filemap_nr_thps_inc(struct address_space *mapping) +{ +#ifdef CONFIG_READ_ONLY_THP_FOR_FS + if (!mapping_thp_support(mapping)) + atomic_inc(&mapping->nr_thps); +#else + WARN_ON_ONCE(1); +#endif +} + +static inline void filemap_nr_thps_dec(struct address_space *mapping) +{ +#ifdef CONFIG_READ_ONLY_THP_FOR_FS + if (!mapping_thp_support(mapping)) + atomic_dec(&mapping->nr_thps); +#else + WARN_ON_ONCE(1); +#endif +} + void release_pages(struct page **pages, int nr); /* From patchwork Wed Jun 10 20:13:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598869 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 25179618 for ; Wed, 10 Jun 2020 20:40:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E5710206A4 for ; Wed, 10 Jun 2020 20:40:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="cVDOzoDr" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E5710206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B8EC98D0036; Wed, 10 Jun 2020 16:39:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id ACA9C8D0033; Wed, 10 Jun 2020 16:39:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B6548D0036; Wed, 10 Jun 2020 16:39:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id 645358D0033 for ; Wed, 10 Jun 2020 16:39:38 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id DF813181ABEA4 for ; Wed, 10 Jun 2020 20:39:37 +0000 (UTC) X-FDA: 76914467994.06.part15_100a84126dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id A59BD1003BAC2 for ; Wed, 10 Jun 2020 20:39:37 +0000 (UTC) X-Spam-Summary: 2,0,0,4caa95b2210e6b24,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2198:2199:2282:2393:2559:2562:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:4321:4384:4605:5007:6117:6261:6653:7576:7903:8603:9036:10004:11026:11232:11473:11658:11914:12043:12296:12297:12438:12555:12895:12986:13894:14096:14181:14394:14721:21080:21433:21451:21627:21990:30003:30034:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: part15_100a84126dce X-Filterd-Recvd-Size: 5580 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=yMWLGdEUMEbWafQbsRKI1zEZCEWuD9YUr6WaHFxn6IQ=; b=cVDOzoDrAmsb1onyp7BR50lT68 oMB3+dTBUmFoeExrfbSgB2Yb/alAaTwsc2mme5JkGTAXZ6JqBX1zisc84UNZcb3YAnONigrVezwwi 0pOH5y/XaesjYgPXrDFhUX5u9fe8kss7e+UUMfDu0rdX+yhOB+Jcot3Q8VJ1nv1CiImh0Qoi6zCrH En6C0w2XZgyY2ZSh1ymzqxfZ/VZKdKBxJBLVYj/FYoYcqAoESErVVTKC0oya4sM7wmNKD2gkWhcV+ iaMR3JBjzV9LSNmQ+bZAiNetkf1YM/TVJKz3LVQ1CRFZo6VqmfmD0tipiekAfxQxWiJGRDdJTjfpP hdtaLNlA==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003UX-SQ; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Hellwig Subject: [PATCH v6 16/51] fs: Introduce i_blocks_per_page Date: Wed, 10 Jun 2020 13:13:10 -0700 Message-Id: <20200610201345.13273-17-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: A59BD1003BAC2 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" This helper is useful for both THPs and for supporting block size larger than page size. Convert some example users (we have a few different ways of writing this idiom). Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- fs/iomap/buffered-io.c | 8 ++++---- fs/jfs/jfs_metapage.c | 2 +- fs/xfs/xfs_aops.c | 2 +- include/linux/pagemap.h | 15 +++++++++++++++ 4 files changed, 21 insertions(+), 6 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index a1ed7620fbac..4d3d0abc1421 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -46,7 +46,7 @@ iomap_page_create(struct inode *inode, struct page *page) { struct iomap_page *iop = to_iomap_page(page); - if (iop || i_blocksize(inode) == PAGE_SIZE) + if (iop || i_blocks_per_page(inode, page) <= 1) return iop; iop = kmalloc(sizeof(*iop), GFP_NOFS | __GFP_NOFAIL); @@ -147,7 +147,7 @@ iomap_iop_set_range_uptodate(struct page *page, unsigned off, unsigned len) unsigned int i; spin_lock_irqsave(&iop->uptodate_lock, flags); - for (i = 0; i < PAGE_SIZE / i_blocksize(inode); i++) { + for (i = 0; i < i_blocks_per_page(inode, page); i++) { if (i >= first && i <= last) set_bit(i, iop->uptodate); else if (!test_bit(i, iop->uptodate)) @@ -1079,7 +1079,7 @@ iomap_finish_page_writeback(struct inode *inode, struct page *page, mapping_set_error(inode->i_mapping, -EIO); } - WARN_ON_ONCE(i_blocksize(inode) < PAGE_SIZE && !iop); + WARN_ON_ONCE(i_blocks_per_page(inode, page) > 1 && !iop); WARN_ON_ONCE(iop && atomic_read(&iop->write_count) <= 0); if (!iop || atomic_dec_and_test(&iop->write_count)) @@ -1375,7 +1375,7 @@ iomap_writepage_map(struct iomap_writepage_ctx *wpc, int error = 0, count = 0, i; LIST_HEAD(submit_list); - WARN_ON_ONCE(i_blocksize(inode) < PAGE_SIZE && !iop); + WARN_ON_ONCE(i_blocks_per_page(inode, page) > 1 && !iop); WARN_ON_ONCE(iop && atomic_read(&iop->write_count) != 0); /* diff --git a/fs/jfs/jfs_metapage.c b/fs/jfs/jfs_metapage.c index a2f5338a5ea1..176580f54af9 100644 --- a/fs/jfs/jfs_metapage.c +++ b/fs/jfs/jfs_metapage.c @@ -473,7 +473,7 @@ static int metapage_readpage(struct file *fp, struct page *page) struct inode *inode = page->mapping->host; struct bio *bio = NULL; int block_offset; - int blocks_per_page = PAGE_SIZE >> inode->i_blkbits; + int blocks_per_page = i_blocks_per_page(inode, page); sector_t page_start; /* address of page in fs blocks */ sector_t pblock; int xlen; diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index b35611882ff9..55d126d4e096 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -544,7 +544,7 @@ xfs_discard_page( page, ip->i_ino, offset); error = xfs_bmap_punch_delalloc_range(ip, start_fsb, - PAGE_SIZE / i_blocksize(inode)); + i_blocks_per_page(inode, page)); if (error && !XFS_FORCED_SHUTDOWN(mp)) xfs_alert(mp, "page discard unable to remove delalloc mapping."); out_invalidate: diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 644caff3e78b..3ad0c92be649 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -891,4 +891,19 @@ static inline int page_mkwrite_check_truncate(struct page *page, return offset; } +/** + * i_blocks_per_page - How many blocks fit in this page. + * @inode: The inode which contains the blocks. + * @page: The page (head page if the page is a THP). + * + * If the block size is larger than the size of this page, return zero. + * + * Context: Any context. + * Return: The number of filesystem blocks covered by this page. + */ +static inline +unsigned int i_blocks_per_page(struct inode *inode, struct page *page) +{ + return thp_size(page) >> inode->i_blkbits; +} #endif /* _LINUX_PAGEMAP_H */ From patchwork Wed Jun 10 20:13:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598677 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7B8C114B7 for ; Wed, 10 Jun 2020 20:14:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 481DB20848 for ; Wed, 10 Jun 2020 20:14:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="PJFR2z3i" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 481DB20848 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2F4D48D0011; Wed, 10 Jun 2020 16:13:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1AD398D000E; Wed, 10 Jun 2020 16:13:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCFCC8D0012; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0160.hostedemail.com [216.40.44.160]) by kanga.kvack.org (Postfix) with ESMTP id AE7B38D000E for ; Wed, 10 Jun 2020 16:13:52 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6393F1A27C2 for ; Wed, 10 Jun 2020 20:13:52 +0000 (UTC) X-FDA: 76914403104.17.nut44_371480726dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 37960180D0E90 for ; Wed, 10 Jun 2020 20:13:52 +0000 (UTC) X-Spam-Summary: 2,0,0,ff29c638ff075a00,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:421:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3871:4605:5007:6261:6630:6653:7576:8603:9036:10004:11026:11658:11914:12043:12114:12297:12438:12555:12895:13069:13161:13229:13311:13357:13894:14096:14181:14384:14394:14721:21080:21451:21627:21990:30034:30054:30070,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: nut44_371480726dce X-Filterd-Recvd-Size: 3164 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=4figGncYPDs23Of7IYih6m6FXzxa/L1ppRPiwe0GXZ0=; b=PJFR2z3iqH2Nw5qEV5yTriVnR4 HHwB6hGy6N6yjAOR3migbMhFuAjDYBO6zIYCw/fNrcXicSxzoCBkGdnPHKmUgvpT3T7TDaLRyvaav JGyWQdEtV3qbBb8Akw6Oicgk1HRqEMSDKJxUuBiCo2XzUryuxUSF9ai14FYO7HMb5E4/Zviwb/xGA S7OwUAUqIjL/lz0EN8hcdX7VP/Url6U4TE9XF7AVlkVigU2+dimH6OoHYtq/2FFpNUe1FupIPkIOg 2atAbD5fBVz0PhB1Bm/qWf0zS8QyDPlrlJg1vM2oxmuuHtC7d1wQRZUWYh/wEa8J9je9BFvpqk0R4 ceJa2iyQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003Ue-TO; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 17/51] fs: Make page_mkwrite_check_truncate thp-aware Date: Wed, 10 Jun 2020 13:13:11 -0700 Message-Id: <20200610201345.13273-18-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 37960180D0E90 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" If the page is compound, check the last index in the page and return the appropriate size. Change the return type to ssize_t in case we ever support pages larger than 2GB. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 3ad0c92be649..f47ba9f18f3e 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -868,22 +868,22 @@ static inline unsigned long dir_pages(struct inode *inode) * @page: the page to check * @inode: the inode to check the page against * - * Returns the number of bytes in the page up to EOF, + * Return: The number of bytes in the page up to EOF, * or -EFAULT if the page was truncated. */ -static inline int page_mkwrite_check_truncate(struct page *page, +static inline ssize_t page_mkwrite_check_truncate(struct page *page, struct inode *inode) { loff_t size = i_size_read(inode); pgoff_t index = size >> PAGE_SHIFT; - int offset = offset_in_page(size); + unsigned long offset = offset_in_thp(page, size); if (page->mapping != inode->i_mapping) return -EFAULT; /* page is wholly inside EOF */ - if (page->index < index) - return PAGE_SIZE; + if (page->index + thp_nr_pages(page) - 1 < index) + return thp_size(page); /* page is wholly past EOF */ if (page->index > index || !offset) return -EFAULT; From patchwork Wed Jun 10 20:13:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598681 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2D8D014B7 for ; Wed, 10 Jun 2020 20:14:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EC28120820 for ; Wed, 10 Jun 2020 20:14:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="HoAnYGt/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EC28120820 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F3BF18D0013; Wed, 10 Jun 2020 16:13:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EF1118D000E; Wed, 10 Jun 2020 16:13:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB1AB8D0013; Wed, 10 Jun 2020 16:13:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0006.hostedemail.com [216.40.44.6]) by kanga.kvack.org (Postfix) with ESMTP id C3ABB8D000E for ; Wed, 10 Jun 2020 16:13:54 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8469B8013B2D for ; Wed, 10 Jun 2020 20:13:54 +0000 (UTC) X-FDA: 76914403188.03.door87_231000926dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 515F31359D for ; Wed, 10 Jun 2020 20:13:54 +0000 (UTC) X-Spam-Summary: 2,0,0,5b1f0866eaead1d6,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2393:2559:2562:2898:3138:3139:3140:3141:3142:3355:3743:3866:3867:3868:3870:3871:3872:4250:4321:4605:5007:6117:6119:6261:6653:7576:7875:7901:9592:10004:10946:11026:11232:11473:11658:11914:12043:12291:12297:12438:12555:12683:12895:12986:13894:14096:14181:14394:14721:21080:21451:21627:30054:30070,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:0,LUA_SUMMARY:none X-HE-Tag: door87_231000926dce X-Filterd-Recvd-Size: 5023 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=5TnSyW1Zm44ghLTpiBh1vxVIfpAZiM8VRVnGQzkBos0=; b=HoAnYGt/Qu5AdQzbQrAIYqUMJi S65+OguqCX6m+LtbZNviyhLWcfjuK2yg8Jm2YDpYlFi9ZsaboH4MWW7atwRUtvermomPgj9uKul0E UIk2S3YAk3keYTFtdRm3+nAIZBlt7WzGvRdl4HF9lNWiWwbGou/8huNYl8xfN35fg8fVC8lNBSGKG 9yUFSRc+bwpCmiROjqFot5CRXDcwinRTCfWvyDrJ1PuhiMnyo1bgNqnSUMk+rG8ca/ZRzU6d8s/mE osmjA58+XEj9hHMuruLNCtn7GxkRnB5Iu9bdZFDaih6HQ4zas9EgJBav4CMdC14I7+/d8dTeXZDi9 UXNDk8Ag==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003Ui-UY; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 18/51] mm: Support THPs in zero_user_segments Date: Wed, 10 Jun 2020 13:13:12 -0700 Message-Id: <20200610201345.13273-19-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 515F31359D X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" We can only kmap() one subpage of a THP at a time, so loop over all relevant subpages, skipping ones which don't need to be zeroed. This is too large to inline when THPs are enabled and we actually need highmem, so put it in highmem.c. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/highmem.h | 11 ++++++-- mm/highmem.c | 62 +++++++++++++++++++++++++++++++++++++++-- 2 files changed, 68 insertions(+), 5 deletions(-) diff --git a/include/linux/highmem.h b/include/linux/highmem.h index d6e82e3de027..f05589513103 100644 --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -284,13 +284,17 @@ static inline void clear_highpage(struct page *page) kunmap_atomic(kaddr); } +#if defined(CONFIG_HIGHMEM) && defined(CONFIG_TRANSPARENT_HUGEPAGE) +void zero_user_segments(struct page *page, unsigned start1, unsigned end1, + unsigned start2, unsigned end2); +#else /* !HIGHMEM || !TRANSPARENT_HUGEPAGE */ static inline void zero_user_segments(struct page *page, - unsigned start1, unsigned end1, - unsigned start2, unsigned end2) + unsigned start1, unsigned end1, + unsigned start2, unsigned end2) { void *kaddr = kmap_atomic(page); - BUG_ON(end1 > PAGE_SIZE || end2 > PAGE_SIZE); + BUG_ON(end1 > thp_size(page) || end2 > thp_size(page)); if (end1 > start1) memset(kaddr + start1, 0, end1 - start1); @@ -301,6 +305,7 @@ static inline void zero_user_segments(struct page *page, kunmap_atomic(kaddr); flush_dcache_page(page); } +#endif /* !HIGHMEM || !TRANSPARENT_HUGEPAGE */ static inline void zero_user_segment(struct page *page, unsigned start, unsigned end) diff --git a/mm/highmem.c b/mm/highmem.c index 64d8dea47dd1..686cae2f1ba5 100644 --- a/mm/highmem.c +++ b/mm/highmem.c @@ -367,9 +367,67 @@ void kunmap_high(struct page *page) if (need_wakeup) wake_up(pkmap_map_wait); } - EXPORT_SYMBOL(kunmap_high); -#endif + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +void zero_user_segments(struct page *page, unsigned start1, unsigned end1, + unsigned start2, unsigned end2) +{ + unsigned int i; + + BUG_ON(end1 > thp_size(page) || end2 > thp_size(page)); + + for (i = 0; i < thp_nr_pages(page); i++) { + void *kaddr; + unsigned this_end; + + if (end1 == 0 && start2 >= PAGE_SIZE) { + start2 -= PAGE_SIZE; + end2 -= PAGE_SIZE; + continue; + } + + if (start1 >= PAGE_SIZE) { + start1 -= PAGE_SIZE; + end1 -= PAGE_SIZE; + if (start2) { + start2 -= PAGE_SIZE; + end2 -= PAGE_SIZE; + } + continue; + } + + kaddr = kmap_atomic(page + i); + + this_end = min_t(unsigned, end1, PAGE_SIZE); + if (end1 > start1) + memset(kaddr + start1, 0, this_end - start1); + end1 -= this_end; + start1 = 0; + + if (start2 >= PAGE_SIZE) { + start2 -= PAGE_SIZE; + end2 -= PAGE_SIZE; + } else { + this_end = min_t(unsigned, end2, PAGE_SIZE); + if (end2 > start2) + memset(kaddr + start2, 0, this_end - start2); + end2 -= this_end; + start2 = 0; + } + + kunmap_atomic(kaddr); + + if (!end1 && !end2) + break; + } + flush_dcache_page(page); + + BUG_ON((start1 | start2 | end1 | end2) != 0); +} +EXPORT_SYMBOL(zero_user_segments); +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_HIGHMEM */ #if defined(HASHED_PAGE_VIRTUAL) From patchwork Wed Jun 10 20:13:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598661 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DA7B814B7 for ; Wed, 10 Jun 2020 20:14:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9A06F2074B for ; Wed, 10 Jun 2020 20:14:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Ib0/cbiZ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9A06F2074B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6D7528D000D; Wed, 10 Jun 2020 16:13:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 686568D0009; Wed, 10 Jun 2020 16:13:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 526948D000D; Wed, 10 Jun 2020 16:13:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0084.hostedemail.com [216.40.44.84]) by kanga.kvack.org (Postfix) with ESMTP id 34AEE8D000C for ; Wed, 10 Jun 2020 16:13:51 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id DF876181ABEA4 for ; Wed, 10 Jun 2020 20:13:50 +0000 (UTC) X-FDA: 76914403020.21.crook55_111447c26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id B6690180442ED for ; Wed, 10 Jun 2020 20:13:50 +0000 (UTC) X-Spam-Summary: 2,0,0,0f75acf0b562c8ae,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3867:3868:4321:5007:6261:6653:7576:7875:10004:11026:11658:11914:12043:12296:12297:12438:12555:12895:12986:13069:13161:13229:13311:13357:13894:14096:14181:14384:14394:14721:21080:21324:21451:21627:21796:30036:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: crook55_111447c26dce X-Filterd-Recvd-Size: 3501 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf09.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=WueiR9ClBR+SuUmlwhBVixkLUapU9LiFVjs2FwKptFw=; b=Ib0/cbiZsMjrh5GZh4ZA8ZsdY2 klpGZJeArXDdQfA2xiQwLfZNtsV95uw+Q/RHNtGh8hpAOnotLkrSmMge24xAWMgPIZL9d6YQhdPIF P05camGYRlTJ1tZNEhJ4tv4nbwD86PMeP30wQ5SAxEZrgJSK68d9usY7lJFV1jiUEYpMen39soRQ4 yR7pcyiXFKgjgoumY2NiTcWzA/WgduKS2tg0ud4bV5A3+m9xmSoC66ciYwGftMtfGfx35qtwXGOSN 69rSoh0QIvooYftJKD2LAW91mhgPhd9WJGlRvA8pfxm/loFcLgvSh23dHahH1GVXhnmmrSZJHqY6C nPY9qiUA==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76Z-0003Up-Vj; Wed, 10 Jun 2020 20:13:47 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 19/51] mm: Zero the head page, not the tail page Date: Wed, 10 Jun 2020 13:13:13 -0700 Message-Id: <20200610201345.13273-20-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: B6690180442ED X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Pass the head page to zero_user_segment(), not the tail page, and adjust the byte offsets appropriately. Signed-off-by: Matthew Wilcox (Oracle) --- mm/shmem.c | 7 +++++++ mm/truncate.c | 7 +++++++ 2 files changed, 14 insertions(+) diff --git a/mm/shmem.c b/mm/shmem.c index a05d129a45e9..55405d811cfd 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -898,11 +898,18 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, struct page *page = NULL; shmem_getpage(inode, start - 1, &page, SGP_READ); if (page) { + struct page *head = thp_head(page); unsigned int top = PAGE_SIZE; if (start > end) { top = partial_end; partial_end = 0; } + if (head != page) { + unsigned int diff = start - 1 - head->index; + partial_start += diff << PAGE_SHIFT; + top += diff << PAGE_SHIFT; + page = head; + } zero_user_segment(page, partial_start, top); set_page_dirty(page); unlock_page(page); diff --git a/mm/truncate.c b/mm/truncate.c index dd9ebc1da356..152974888124 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -374,12 +374,19 @@ void truncate_inode_pages_range(struct address_space *mapping, if (partial_start) { struct page *page = find_lock_page(mapping, start - 1); if (page) { + struct page *head = thp_head(page); unsigned int top = PAGE_SIZE; if (start > end) { /* Truncation within a single page */ top = partial_end; partial_end = 0; } + if (head != page) { + unsigned int diff = start - 1 - head->index; + partial_start += diff << PAGE_SHIFT; + top += diff << PAGE_SHIFT; + page = head; + } wait_on_page_writeback(page); zero_user_segment(page, partial_start, top); cleancache_invalidate_page(mapping, page); From patchwork Wed Jun 10 20:13:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598663 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A28EC138C for ; Wed, 10 Jun 2020 20:14:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6F1D72074B for ; Wed, 10 Jun 2020 20:14:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="GALygLUc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6F1D72074B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 95FA98D0009; Wed, 10 Jun 2020 16:13:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7C22B8D000E; Wed, 10 Jun 2020 16:13:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C3D58D000C; Wed, 10 Jun 2020 16:13:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0129.hostedemail.com [216.40.44.129]) by kanga.kvack.org (Postfix) with ESMTP id 28DA88D0009 for ; Wed, 10 Jun 2020 16:13:51 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DA1C01A1E3D for ; Wed, 10 Jun 2020 20:13:50 +0000 (UTC) X-FDA: 76914403020.12.smile51_020ca8b26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 9BD30180B69CB for ; Wed, 10 Jun 2020 20:13:50 +0000 (UTC) X-Spam-Summary: 2,0,0,c5c3a1c1d9e53ee6,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2393:2553:2559:2562:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3872:4250:4321:5007:6119:6261:6653:6691:7576:7875:10004:11026:11473:11658:11914:12043:12114:12291:12296:12297:12438:12555:12895:13894:14096:14181:14394:14721:21080:21451:21524:21627:21990:30054:30090,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: smile51_020ca8b26dce X-Filterd-Recvd-Size: 3909 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:13:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=Nm4e+jFJPwTorBmgbzzcocpiWsS+VJ//sRHdCySBlWE=; b=GALygLUc1KauPQ2HJMEuNvYE6Q ce7q96xbsj5KD/sjyR8fyip6MN1E2b723YDlEl0u/ZS0CJHj0iP7YyI1b2b0Anr19eipwud+IEFNb R7H8dTDxB70aead3m4fp2UDTZ3r9n+dEazPZKpm1n6QP4gTCj4icZJyV8Q5cqI4UZEXVo1J4qMbi/ XyPDOppf6DtTLTmqSljpz/ay7jeXhBuFZmw49sfZotYJj2T5YRFTbjP/ftsFAg8vbEAmdXirFzLOR U3Qi0lDelVVIBV5QGh48CcHyHcHLEBGoe9eDU5HV/jdkWyebQPjO0ZA4f5xX/X9LCtEZ/qf39aKiW BiF/N9Dw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003Uw-0V; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 20/51] block: Add bio_for_each_thp_segment_all Date: Wed, 10 Jun 2020 13:13:14 -0700 Message-Id: <20200610201345.13273-21-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9BD30180B69CB X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Iterate once for each THP instead of once for each base page. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/bio.h | 13 +++++++++++++ include/linux/bvec.h | 23 +++++++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/include/linux/bio.h b/include/linux/bio.h index 91676d4b2dfe..1489e196abf5 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -131,12 +131,25 @@ static inline bool bio_next_segment(const struct bio *bio, return true; } +static inline bool bio_next_thp_segment(const struct bio *bio, + struct bvec_iter_all *iter) +{ + if (iter->idx >= bio->bi_vcnt) + return false; + + bvec_thp_advance(&bio->bi_io_vec[iter->idx], iter); + return true; +} + /* * drivers should _never_ use the all version - the bio may have been split * before it got to the driver and the driver won't own all of it */ #define bio_for_each_segment_all(bvl, bio, iter) \ for (bvl = bvec_init_iter_all(&iter); bio_next_segment((bio), &iter); ) +#define bio_for_each_thp_segment_all(bvl, bio, iter) \ + for (bvl = bvec_init_iter_all(&iter); \ + bio_next_thp_segment((bio), &iter); ) static inline void bio_advance_iter(const struct bio *bio, struct bvec_iter *iter, unsigned int bytes) diff --git a/include/linux/bvec.h b/include/linux/bvec.h index ac0c7299d5b8..71b435e573e1 100644 --- a/include/linux/bvec.h +++ b/include/linux/bvec.h @@ -162,4 +162,27 @@ static inline void bvec_advance(const struct bio_vec *bvec, } } +static inline void bvec_thp_advance(const struct bio_vec *bvec, + struct bvec_iter_all *iter_all) +{ + struct bio_vec *bv = &iter_all->bv; + unsigned int page_size = thp_size(bvec->bv_page); + + if (iter_all->done) { + bv->bv_page += thp_nr_pages(bv->bv_page); + bv->bv_offset = 0; + } else { + BUG_ON(bvec->bv_offset >= page_size); + bv->bv_page = bvec->bv_page; + bv->bv_offset = bvec->bv_offset & (page_size - 1); + } + bv->bv_len = min(page_size - bv->bv_offset, + bvec->bv_len - iter_all->done); + iter_all->done += bv->bv_len; + + if (iter_all->done == bvec->bv_len) { + iter_all->idx++; + iter_all->done = 0; + } +} #endif /* __LINUX_BVEC_ITER_H */ From patchwork Wed Jun 10 20:13:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598851 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6874B618 for ; Wed, 10 Jun 2020 20:39:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 34B56206A4 for ; Wed, 10 Jun 2020 20:39:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="WJlfH1qc" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 34B56206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9216D8D002E; Wed, 10 Jun 2020 16:39:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8A78F8D002D; Wed, 10 Jun 2020 16:39:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 744A28D002E; Wed, 10 Jun 2020 16:39:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id 5600D8D002D for ; Wed, 10 Jun 2020 16:39:30 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1C7B9801ABF9 for ; Wed, 10 Jun 2020 20:39:30 +0000 (UTC) X-FDA: 76914467700.24.time83_5b048a926dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id C7F113E70E for ; Wed, 10 Jun 2020 20:39:29 +0000 (UTC) X-Spam-Summary: 2,0,0,3045e0f29918be18,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:3870:3874:5007:6261:6653:7576:7901:7903:10004:11026:11658:11914:12043:12296:12297:12438:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14721:14824:21080:21451:21627:21990:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: time83_5b048a926dce X-Filterd-Recvd-Size: 2629 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=78nHOizw+EWRinzuA/mQKnq3TFssCLkrB+hP2+3ZTSU=; b=WJlfH1qcHthNYhpNssG2ts7bW9 ZLR7XRa3dIHv+Zrs2yb31eOEQQDU8OYKVvYwX+ul3o9tt2oar2VaK/IUaYfLTwGOeDPDBS0Bgk9gc GLOpvCF0XyxW2jHTOHLld13d8x8nYvD0TCGu+OW9CIfjXDTYt0DL6+YI9H5/EKl7bq+OvXWCjAdDM YkB7wbFtYm2K+vhYxWotrB39+oYIqp9Yl7DkUVfDjdZtU0D5b0Fn6pklltYmVUrXkFUOHD5vE66If dwKqAsKDkcD+Sy/obPNTgjnXO7Ix9OhrO4AsDvcxKcKkdeb5czPcyebiLYQTEtNM7VswwxqtwEJlu TC0ffTxQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003V3-1h; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 21/51] block: Support THPs in page_is_mergeable Date: Wed, 10 Jun 2020 13:13:15 -0700 Message-Id: <20200610201345.13273-22-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C7F113E70E X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" page_is_mergeable() would incorrectly claim that two IOs were on different pages because they were on different base pages rather than on the same THP. This led to a reference counting bug in iomap. Simplify the 'same_page' test by just comparing whether we have the same struct page instead of doing arithmetic on the physical addresses. Signed-off-by: Matthew Wilcox (Oracle) --- block/bio.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/bio.c b/block/bio.c index 5235da6434aa..cd677cde853d 100644 --- a/block/bio.c +++ b/block/bio.c @@ -747,7 +747,7 @@ static inline bool page_is_mergeable(const struct bio_vec *bv, if (xen_domain() && !xen_biovec_phys_mergeable(bv, page)) return false; - *same_page = ((vec_end_addr & PAGE_MASK) == page_addr); + *same_page = bv->bv_page == page; if (!*same_page && pfn_to_page(PFN_DOWN(vec_end_addr)) + 1 != page) return false; return true; From patchwork Wed Jun 10 20:13:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598843 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7D0F490 for ; Wed, 10 Jun 2020 20:39:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3FCAF206A4 for ; Wed, 10 Jun 2020 20:39:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="KUjUe2Eh" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3FCAF206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B8C818D0029; Wed, 10 Jun 2020 16:39:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AC6C18D002A; Wed, 10 Jun 2020 16:39:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 98CE18D0029; Wed, 10 Jun 2020 16:39:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0217.hostedemail.com [216.40.44.217]) by kanga.kvack.org (Postfix) with ESMTP id 78B528D0028 for ; Wed, 10 Jun 2020 16:39:27 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 3DF9F3E56D for ; Wed, 10 Jun 2020 20:39:27 +0000 (UTC) X-FDA: 76914467574.17.snow47_550221d26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin17.hostedemail.com (Postfix) with ESMTP id 12FA6180D0E90 for ; Wed, 10 Jun 2020 20:39:27 +0000 (UTC) X-Spam-Summary: 2,0,0,de3a0e2d9986845e,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2198:2199:2282:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:3868:3870:3871:3874:4321:5007:6261:6653:7576:7875:8603:8957:9036:9592:10004:11026:11232:11473:11658:11914:12043:12296:12297:12438:12555:12895:13894:14096:14181:14394:14721:21080:21451:21627:21990:30046:30054:30070,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:0,LUA_SUMMARY:none X-HE-Tag: snow47_550221d26dce X-Filterd-Recvd-Size: 4234 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=RjoGa99XdcmVsHZZNtbX8kRo95MxAX4SaxcEuGmYN1E=; b=KUjUe2EhH1YNoqavI3yyeilX1v AIg9ZnYh349tXQaVUhDpoG2x1xeIRoPraIhW1SltAfmuALJ/GOA1nB+f5BrXqXIFvS3lzpwzB0Zif 7qimYe6y4IaiDQQPu+e60gSu+BJhi9oGwaO7UsGAFUSPiEUGSK02SyWrw4cBI8OrS5u3jvMrKiJ5X qo4bzCcQ5JLAJ16cEp3cBISjz6gk9X95Tc4XH75Uvs7YLr85Toyw6tEjPGrJN5OEQgvvx9KhyV/D/ DQhnNlURXvhxqU+2k/m5VMD72BPFikRsn4xKcWVD3ixS37XhL8hFUta9Y+gDVwnwtTKB2WN9E/7QQ +IxgPI8Q==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003V7-2z; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 22/51] iomap: Support arbitrarily many blocks per page Date: Wed, 10 Jun 2020 13:13:16 -0700 Message-Id: <20200610201345.13273-23-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 12FA6180D0E90 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Size the uptodate array dynamically. Now that this array is protected by a spinlock, we can use bitmap functions to set the bits in this array instead of a loop around set_bit(). Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 25 ++++++++----------------- 1 file changed, 8 insertions(+), 17 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 4d3d0abc1421..1a398af42ed2 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -22,14 +22,14 @@ #include "../internal.h" /* - * Structure allocated for each page when block size < PAGE_SIZE to track + * Structure allocated for each page when block size < page size to track * sub-page uptodate status and I/O completions. */ struct iomap_page { atomic_t read_count; atomic_t write_count; spinlock_t uptodate_lock; - DECLARE_BITMAP(uptodate, PAGE_SIZE / 512); + unsigned long uptodate[]; }; static inline struct iomap_page *to_iomap_page(struct page *page) @@ -45,15 +45,14 @@ static struct iomap_page * iomap_page_create(struct inode *inode, struct page *page) { struct iomap_page *iop = to_iomap_page(page); + unsigned int nr_blocks = i_blocks_per_page(inode, page); - if (iop || i_blocks_per_page(inode, page) <= 1) + if (iop || nr_blocks <= 1) return iop; - iop = kmalloc(sizeof(*iop), GFP_NOFS | __GFP_NOFAIL); - atomic_set(&iop->read_count, 0); - atomic_set(&iop->write_count, 0); + iop = kzalloc(struct_size(iop, uptodate, BITS_TO_LONGS(nr_blocks)), + GFP_NOFS | __GFP_NOFAIL); spin_lock_init(&iop->uptodate_lock); - bitmap_zero(iop->uptodate, PAGE_SIZE / SECTOR_SIZE); /* * migrate_page_move_mapping() assumes that pages with private data have @@ -142,19 +141,11 @@ iomap_iop_set_range_uptodate(struct page *page, unsigned off, unsigned len) struct inode *inode = page->mapping->host; unsigned first = off >> inode->i_blkbits; unsigned last = (off + len - 1) >> inode->i_blkbits; - bool uptodate = true; unsigned long flags; - unsigned int i; spin_lock_irqsave(&iop->uptodate_lock, flags); - for (i = 0; i < i_blocks_per_page(inode, page); i++) { - if (i >= first && i <= last) - set_bit(i, iop->uptodate); - else if (!test_bit(i, iop->uptodate)) - uptodate = false; - } - - if (uptodate) + bitmap_set(iop->uptodate, first, last - first + 1); + if (bitmap_full(iop->uptodate, i_blocks_per_page(inode, page))) SetPageUptodate(page); spin_unlock_irqrestore(&iop->uptodate_lock, flags); } From patchwork Wed Jun 10 20:13:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598847 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0530690 for ; Wed, 10 Jun 2020 20:39:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C5C28206A4 for ; Wed, 10 Jun 2020 20:39:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="aAG6i/6K" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C5C28206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E27038D002B; Wed, 10 Jun 2020 16:39:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DD6058D0028; Wed, 10 Jun 2020 16:39:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7DB98D002B; Wed, 10 Jun 2020 16:39:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0157.hostedemail.com [216.40.44.157]) by kanga.kvack.org (Postfix) with ESMTP id 81FA78D0028 for ; Wed, 10 Jun 2020 16:39:28 -0400 (EDT) Received: from smtpin08.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5496113588 for ; Wed, 10 Jun 2020 20:39:28 +0000 (UTC) X-FDA: 76914467616.08.bead80_540847426dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin08.hostedemail.com (Postfix) with ESMTP id 29C8F1803AA92 for ; Wed, 10 Jun 2020 20:39:28 +0000 (UTC) X-Spam-Summary: 2,0,0,3b432524442300aa,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:69:355:371:372:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:3138:3139:3140:3141:3142:3355:3865:3866:3867:3868:3870:3874:4321:5007:6261:6653:7576:7875:7901:8603:8957:9036:10004:11026:11658:11914:12043:12114:12296:12297:12438:12555:12895:12986:13894:14181:14394:14721:21080:21433:21451:21627:21796:21987:21990:30036:30046:30054:30090,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: bead80_540847426dce X-Filterd-Recvd-Size: 5160 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=VYY9s5PmR1cZwQ4BpRweOnmMFXWZx2Gnlj2etUW5KY8=; b=aAG6i/6K8oKtZYRWYMUiyLnDho 9J9EHOCY5VyIn7O1jBGrPkf1M86kEkutYSUL0J+Ld8Uf5cwemZgdcDh1/s2WJCdvgWE+5J8gnP3sm mqnEWlBV3jYbTnpzzKCHa6R9rHTrJcEmDfNAcHtnLhP4ldRNeDdDDjrAXS88ES5ILGtrdQLSzxN+G BZkjOWgQr7s3Ksqb7X9pqaVZxxIo4WF9NXz/h7d+pEzHU1hBnvAl5khWTrc/woSuTLCzazNHCus5c MTa2Fb89YvW5rmIsGQ7fbww1XVFQ40kgRcFva5/WjgEIpGvKyF0iMu8FPaEWSmrj1G0fTWDNupKE2 +e5AdrnQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003VG-49; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 23/51] iomap: Support THPs in iomap_adjust_read_range Date: Wed, 10 Jun 2020 13:13:17 -0700 Message-Id: <20200610201345.13273-24-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 29C8F1803AA92 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Pass the struct page instead of the iomap_page so we can determine the size of the page. Use offset_in_thp() instead of offset_in_page() and use thp_size() instead of PAGE_SIZE. Convert the arguments to be size_t instead of unsigned int, in case pages ever get larger than 2^31 bytes. Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 1a398af42ed2..1b450e966822 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -77,16 +77,16 @@ iomap_page_release(struct page *page) /* * Calculate the range inside the page that we actually need to read. */ -static void -iomap_adjust_read_range(struct inode *inode, struct iomap_page *iop, - loff_t *pos, loff_t length, unsigned *offp, unsigned *lenp) +static void iomap_adjust_read_range(struct inode *inode, struct page *page, + loff_t *pos, loff_t length, size_t *offp, size_t *lenp) { + struct iomap_page *iop = to_iomap_page(page); loff_t orig_pos = *pos; loff_t isize = i_size_read(inode); unsigned block_bits = inode->i_blkbits; unsigned block_size = (1 << block_bits); - unsigned poff = offset_in_page(*pos); - unsigned plen = min_t(loff_t, PAGE_SIZE - poff, length); + size_t poff = offset_in_thp(page, *pos); + size_t plen = min_t(loff_t, thp_size(page) - poff, length); unsigned first = poff >> block_bits; unsigned last = (poff + plen - 1) >> block_bits; @@ -124,7 +124,7 @@ iomap_adjust_read_range(struct inode *inode, struct iomap_page *iop, * page cache for blocks that are entirely outside of i_size. */ if (orig_pos <= isize && orig_pos + length > isize) { - unsigned end = offset_in_page(isize - 1) >> block_bits; + unsigned end = offset_in_thp(page, isize - 1) >> block_bits; if (first <= end && last > end) plen -= (last - end) * block_size; @@ -241,7 +241,7 @@ iomap_readpage_actor(struct inode *inode, loff_t pos, loff_t length, void *data, struct iomap_page *iop = iomap_page_create(inode, page); bool same_page = false, is_contig = false; loff_t orig_pos = pos; - unsigned poff, plen; + size_t poff, plen; sector_t sector; if (iomap->type == IOMAP_INLINE) { @@ -251,7 +251,7 @@ iomap_readpage_actor(struct inode *inode, loff_t pos, loff_t length, void *data, } /* zero post-eof blocks as the page may be mapped */ - iomap_adjust_read_range(inode, iop, &pos, length, &poff, &plen); + iomap_adjust_read_range(inode, page, &pos, length, &poff, &plen); if (plen == 0) goto done; @@ -560,18 +560,19 @@ static int __iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, int flags, struct page *page, struct iomap *srcmap) { - struct iomap_page *iop = iomap_page_create(inode, page); loff_t block_size = i_blocksize(inode); loff_t block_start = pos & ~(block_size - 1); loff_t block_end = (pos + len + block_size - 1) & ~(block_size - 1); - unsigned from = offset_in_page(pos), to = from + len, poff, plen; + unsigned from = offset_in_page(pos), to = from + len; + size_t poff, plen; int status; if (PageUptodate(page)) return 0; + iomap_page_create(inode, page); do { - iomap_adjust_read_range(inode, iop, &block_start, + iomap_adjust_read_range(inode, page, &block_start, block_end - block_start, &poff, &plen); if (plen == 0) break; From patchwork Wed Jun 10 20:13:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598839 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A46FE618 for ; Wed, 10 Jun 2020 20:39:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 670B8206A4 for ; Wed, 10 Jun 2020 20:39:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="LCORptvi" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 670B8206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 993648D0027; Wed, 10 Jun 2020 16:39:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8CB558D0028; Wed, 10 Jun 2020 16:39:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 74F5B8D000E; Wed, 10 Jun 2020 16:39:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0140.hostedemail.com [216.40.44.140]) by kanga.kvack.org (Postfix) with ESMTP id 5C12E8D000E for ; Wed, 10 Jun 2020 16:39:26 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 191AB80178CB for ; Wed, 10 Jun 2020 20:39:26 +0000 (UTC) X-FDA: 76914467532.04.leg79_130996d26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id EC6498021550 for ; Wed, 10 Jun 2020 20:39:25 +0000 (UTC) X-Spam-Summary: 2,0,0,2ab36fea01a3693a,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3871:3872:3874:4250:5007:6119:6261:6653:7576:7901:8957:10004:11026:11232:11473:11658:11914:12043:12296:12297:12438:12555:12895:13894:14096:14181:14394:14721:21080:21451:21627:21987:30045:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: leg79_130996d26dce X-Filterd-Recvd-Size: 4207 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf45.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=UDxDIbJeI06u3N6GB6RtskaCIa6/f+0o49vLLz1oNrw=; b=LCORptvixsIn242sHeS/5VHXpO 9cijUpAt69pL3N/zjLExsd1abDJ7gozi1ZMubtZlJm0x5DoXnbaIjRPCGRp8Rl3pZKoPgyMYFlxBe FNibNK8EKx6qrIEpe1yxFEgjlQRu/D+xEaRjWH7CLIMwWRCE7v41fp39rEc+cOBCok5lRyqNBQbGH GJ6i6K2TQ/34krSF0IcT5BCXjGIXe1OE8e0O45/enfa7q8AhKmQSzT1NkloScLhkKbv30ZJIQsIjA VX5lAc7vZJLXk9Fh35YJvT8z4KLVEj6kKDlUKfYMBb8jZvrGSV2plLH7rG+OQyi8qIdd1OWxSQVvp DkDmXpQQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003VM-5k; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 24/51] iomap: Support THPs in invalidatepage Date: Wed, 10 Jun 2020 13:13:18 -0700 Message-Id: <20200610201345.13273-25-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: EC6498021550 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" If we're punching a hole in a THP, we need to remove the per-page iomap data, but not clear the dirty bit from the page, so separate the two conditions. This means that writepage can now come across a page with no iop allocated, so remove the assertions that there is already one, and just create one if there isn't one. Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 1b450e966822..b1a2ab2c7a8d 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -487,17 +487,21 @@ EXPORT_SYMBOL_GPL(iomap_releasepage); void iomap_invalidatepage(struct page *page, unsigned int offset, unsigned int len) { + bool full_page = (offset == 0) && (len == thp_size(page)); trace_iomap_invalidatepage(page->mapping->host, offset, len); /* * If we are invalidating the entire page, clear the dirty state from it * and release it to avoid unnecessary buildup of the LRU. */ - if (offset == 0 && len == PAGE_SIZE) { + if (full_page) { WARN_ON_ONCE(PageWriteback(page)); cancel_dirty_page(page); - iomap_page_release(page); } + + /* Punching a hole in a THP requires releasing the iop */ + if (full_page || thp_order(page) > 0) + iomap_page_release(page); } EXPORT_SYMBOL_GPL(iomap_invalidatepage); @@ -1064,14 +1068,13 @@ static void iomap_finish_page_writeback(struct inode *inode, struct page *page, int error) { - struct iomap_page *iop = to_iomap_page(page); + struct iomap_page *iop = iomap_page_create(inode, page); if (error) { SetPageError(page); mapping_set_error(inode->i_mapping, -EIO); } - WARN_ON_ONCE(i_blocks_per_page(inode, page) > 1 && !iop); WARN_ON_ONCE(iop && atomic_read(&iop->write_count) <= 0); if (!iop || atomic_dec_and_test(&iop->write_count)) @@ -1360,14 +1363,13 @@ iomap_writepage_map(struct iomap_writepage_ctx *wpc, struct writeback_control *wbc, struct inode *inode, struct page *page, u64 end_offset) { - struct iomap_page *iop = to_iomap_page(page); + struct iomap_page *iop = iomap_page_create(inode, page); struct iomap_ioend *ioend, *next; unsigned len = i_blocksize(inode); u64 file_offset; /* file offset of page */ int error = 0, count = 0, i; LIST_HEAD(submit_list); - WARN_ON_ONCE(i_blocks_per_page(inode, page) > 1 && !iop); WARN_ON_ONCE(iop && atomic_read(&iop->write_count) != 0); /* From patchwork Wed Jun 10 20:13:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598835 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CEE9390 for ; Wed, 10 Jun 2020 20:39:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9ACCC206A4 for ; Wed, 10 Jun 2020 20:39:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="qcRuO7qB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9ACCC206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 609618D0025; Wed, 10 Jun 2020 16:39:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5BA048D000E; Wed, 10 Jun 2020 16:39:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45B2F8D0025; Wed, 10 Jun 2020 16:39:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0022.hostedemail.com [216.40.44.22]) by kanga.kvack.org (Postfix) with ESMTP id 28FC08D000E for ; Wed, 10 Jun 2020 16:39:24 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id DF64D147B4B for ; Wed, 10 Jun 2020 20:39:23 +0000 (UTC) X-FDA: 76914467406.03.voice20_1d018f226dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id B43301810F for ; Wed, 10 Jun 2020 20:39:23 +0000 (UTC) X-Spam-Summary: 2,0,0,466d234818e7be8b,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2393:2559:2562:2690:2736:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3871:4321:5007:6119:6261:6653:7576:7875:7901:7903:8603:9036:10004:11026:11232:11473:11658:11914:12043:12296:12297:12438:12555:12895:13894:14096:14181:14394:14721:21080:21451:21524:21627:21990:30034:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:3,LUA_SUMMARY:none X-HE-Tag: voice20_1d018f226dce X-Filterd-Recvd-Size: 4301 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=hfj7nvAJQpdF1gepuLuX+65YmLup2Ovu+X8+8QHyAHw=; b=qcRuO7qBGPf0OdGpDWT27F6hEb rgVvYJxj/KD4udt31sGi07iE6f0yKu5B+dceWQbTsMezUbA9o6+sEUOdyvcGUNqlejxVHcpYhMsud 5KD5ussyhQvJz5DQ0VeUxhaXFqbRXF+J0JNpvjy4dlQwykwGx9d4AePGxnsuQn6HpBehY1J6OnXQJ aFJVI1JC2mw8T2oEp6oIMDtvT5aRMfV7hVpXvPRMFNtaQR0yOXtwKcZgDd+FY+9+txO+HaDBLFeSb s02BSBur5St9bzgC5ZcXVYtkzEgU/czCTXoWoUti27KKSSZrfKCxN+f8wIx513HuqAyxX9ziOBx/j 8fRCVe7w==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003VT-79; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 25/51] iomap: Support THPs in read paths Date: Wed, 10 Jun 2020 13:13:19 -0700 Message-Id: <20200610201345.13273-26-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: B43301810F X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Use thp_size() instead of PAGE_SIZE, offset_in_thp() instead of offset_in_page() and bio_for_each_thp_segment_all(). Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index b1a2ab2c7a8d..555ec90e8b54 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -192,7 +192,7 @@ iomap_read_end_io(struct bio *bio) struct bio_vec *bvec; struct bvec_iter_all iter_all; - bio_for_each_segment_all(bvec, bio, iter_all) + bio_for_each_thp_segment_all(bvec, bio, iter_all) iomap_read_page_end_io(bvec, error); bio_put(bio); } @@ -232,6 +232,16 @@ static inline bool iomap_block_needs_zeroing(struct inode *inode, pos >= i_size_read(inode); } +/* + * Estimate the number of vectors we need based on the current page size; + * if we're wrong we'll end up doing an overly large allocation or needing + * to do a second allocation, neither of which is a big deal. + */ +static unsigned int iomap_nr_vecs(struct page *page, loff_t length) +{ + return (length + thp_size(page) - 1) >> page_shift(page); +} + static loff_t iomap_readpage_actor(struct inode *inode, loff_t pos, loff_t length, void *data, struct iomap *iomap, struct iomap *srcmap) @@ -288,7 +298,7 @@ iomap_readpage_actor(struct inode *inode, loff_t pos, loff_t length, void *data, if (!ctx->bio || !is_contig || bio_full(ctx->bio, plen)) { gfp_t gfp = mapping_gfp_constraint(page->mapping, GFP_KERNEL); gfp_t orig_gfp = gfp; - int nr_vecs = (length + PAGE_SIZE - 1) >> PAGE_SHIFT; + int nr_vecs = iomap_nr_vecs(page, length); if (ctx->bio) submit_bio(ctx->bio); @@ -332,9 +342,9 @@ iomap_readpage(struct page *page, const struct iomap_ops *ops) trace_iomap_readpage(page->mapping->host, 1); - for (poff = 0; poff < PAGE_SIZE; poff += ret) { + for (poff = 0; poff < thp_size(page); poff += ret) { ret = iomap_apply(inode, page_offset(page) + poff, - PAGE_SIZE - poff, 0, ops, &ctx, + thp_size(page) - poff, 0, ops, &ctx, iomap_readpage_actor); if (ret <= 0) { WARN_ON_ONCE(ret == 0); @@ -368,7 +378,8 @@ iomap_readahead_actor(struct inode *inode, loff_t pos, loff_t length, loff_t done, ret; for (done = 0; done < length; done += ret) { - if (ctx->cur_page && offset_in_page(pos + done) == 0) { + if (ctx->cur_page && + offset_in_thp(ctx->cur_page, pos + done) == 0) { if (!ctx->cur_page_in_bio) unlock_page(ctx->cur_page); put_page(ctx->cur_page); From patchwork Wed Jun 10 20:13:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598825 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 447F6618 for ; Wed, 10 Jun 2020 20:39:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0DEBD206A4 for ; Wed, 10 Jun 2020 20:39:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="tXkviFNu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0DEBD206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A03848D0020; Wed, 10 Jun 2020 16:39:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 963DB8D000E; Wed, 10 Jun 2020 16:39:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 806EA8D0020; Wed, 10 Jun 2020 16:39:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0002.hostedemail.com [216.40.44.2]) by kanga.kvack.org (Postfix) with ESMTP id 4F86A8D000E for ; Wed, 10 Jun 2020 16:39:16 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 04B85180AEF50 for ; Wed, 10 Jun 2020 20:39:15 +0000 (UTC) X-FDA: 76914467112.20.rice69_431804d26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id C8FD8182152FD for ; Wed, 10 Jun 2020 20:39:15 +0000 (UTC) X-Spam-Summary: 2,0,0,98edcbe7e241f1bf,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:69:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2393:2553:2559:2562:3138:3139:3140:3141:3142:3354:3865:3866:3867:3868:3870:3872:4321:5007:6261:6630:6653:7576:8603:10004:11026:11473:11658:11914:12043:12296:12297:12438:12555:12683:12895:13894:14096:14181:14394:14721:21080:21433:21451:21627:21990:30034:30046:30054:30070:30090,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: rice69_431804d26dce X-Filterd-Recvd-Size: 4543 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=ugHC/UWmtHSn08iaOdB7fIiqwpchGhJFCvJwrf5/iGA=; b=tXkviFNu+Ol6AvIwDGb5N/BYqR L3m7YF30nlbbilanZVQIxQL3uS7W7tJ2Q6Q+VyoYnn4XWazp6KLwgwx/1zqEA67tMOi1ADizS6rJD l1jYAbsLPda4V48o/1J1XPeGDJDCOSYU78zkUudhBlIvknxNpDvBlxVhSR8bOuBYQ2L96A85SG1+0 dSn6PpzrCu1rlIkwOFWeUVrfe/IOCkaIZyGnChgSM1BBxcvEAZb6MK+0Rqi7CItepKhmhVJI+ZFnd WuJ8eb95wBSyAD/7B1IK0kXQiPO/HDGv/kPjsNooACZ8DrlYWq5seVxIf/ts5ArNm53muftkM0iYd hom2gr1w==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003Vc-8x; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 26/51] iomap: Convert iomap_write_end types Date: Wed, 10 Jun 2020 13:13:20 -0700 Message-Id: <20200610201345.13273-27-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C8FD8182152FD X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" iomap_write_end cannot return an error, so switch it to return size_t instead of int and remove the error checking from the callers. Also convert the arguments to size_t from unsigned int, in case anyone ever wants to support a page size larger than 2GB. Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 555ec90e8b54..fffdf5906e99 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -692,9 +692,8 @@ iomap_set_page_dirty(struct page *page) } EXPORT_SYMBOL_GPL(iomap_set_page_dirty); -static int -__iomap_write_end(struct inode *inode, loff_t pos, unsigned len, - unsigned copied, struct page *page) +static size_t __iomap_write_end(struct inode *inode, loff_t pos, size_t len, + size_t copied, struct page *page) { flush_dcache_page(page); @@ -716,9 +715,8 @@ __iomap_write_end(struct inode *inode, loff_t pos, unsigned len, return copied; } -static int -iomap_write_end_inline(struct inode *inode, struct page *page, - struct iomap *iomap, loff_t pos, unsigned copied) +static size_t iomap_write_end_inline(struct inode *inode, struct page *page, + struct iomap *iomap, loff_t pos, size_t copied) { void *addr; @@ -733,13 +731,14 @@ iomap_write_end_inline(struct inode *inode, struct page *page, return copied; } -static int -iomap_write_end(struct inode *inode, loff_t pos, unsigned len, unsigned copied, - struct page *page, struct iomap *iomap, struct iomap *srcmap) +/* Returns the number of bytes copied. May be 0. Cannot be an errno. */ +static size_t iomap_write_end(struct inode *inode, loff_t pos, size_t len, + size_t copied, struct page *page, struct iomap *iomap, + struct iomap *srcmap) { const struct iomap_page_ops *page_ops = iomap->page_ops; loff_t old_size = inode->i_size; - int ret; + size_t ret; if (srcmap->type == IOMAP_INLINE) { ret = iomap_write_end_inline(inode, page, iomap, pos, copied); @@ -820,11 +819,8 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data, flush_dcache_page(page); - status = iomap_write_end(inode, pos, bytes, copied, page, iomap, + copied = iomap_write_end(inode, pos, bytes, copied, page, iomap, srcmap); - if (unlikely(status < 0)) - break; - copied = status; cond_resched(); @@ -898,11 +894,8 @@ iomap_unshare_actor(struct inode *inode, loff_t pos, loff_t length, void *data, status = iomap_write_end(inode, pos, bytes, bytes, page, iomap, srcmap); - if (unlikely(status <= 0)) { - if (WARN_ON_ONCE(status == 0)) - return -EIO; - return status; - } + if (WARN_ON_ONCE(status == 0)) + return -EIO; cond_resched(); From patchwork Wed Jun 10 20:13:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598837 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD80590 for ; Wed, 10 Jun 2020 20:39:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 899A2206A4 for ; Wed, 10 Jun 2020 20:39:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="f5bKWcgH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 899A2206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9B9D88D0026; Wed, 10 Jun 2020 16:39:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8A9A58D000E; Wed, 10 Jun 2020 16:39:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D3D88D0026; Wed, 10 Jun 2020 16:39:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0149.hostedemail.com [216.40.44.149]) by kanga.kvack.org (Postfix) with ESMTP id 4BE4B8D000E for ; Wed, 10 Jun 2020 16:39:25 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 18D8811AC18 for ; Wed, 10 Jun 2020 20:39:25 +0000 (UTC) X-FDA: 76914467490.29.plane38_63147a726dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id E327B18086E22 for ; Wed, 10 Jun 2020 20:39:24 +0000 (UTC) X-Spam-Summary: 2,0,0,236125f0adcff331,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:69:355:371:372:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1605:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3865:3866:3867:3868:3871:3872:4117:4250:4321:4605:5007:6261:6653:7576:7875:8603:9036:9592:10004:11026:11473:11658:11914:12043:12296:12297:12438:12555:12683:12895:12986:13161:13229:13894:14096:14181:14394:14721:21080:21212:21451:21627:21740:21796:21990:30036:30046:30054:30070:30080,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:19,LUA_SUMMARY:none X-HE-Tag: plane38_63147a726dce X-Filterd-Recvd-Size: 6493 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=uGAtnGn+n3Jc24Na5BRkCaH/Nj1ojalZsrf8LfvpEMU=; b=f5bKWcgHXW6o+ZzSODjgwHECEx kwYqPQcdX6oIPqpA9AwddVLy3KemXyT6U7H9Eg2T6tgrdQzfYt3ELn//qswyYA2t2iLPZoZ0JRwf8 T9/6oztx0bbqmdTxt6LAimS9X662A3SjVQg2hFdPOYkuo1LX7BtqF9KgZuA80RH7LUcGgox7apGtd kcaKvJk1QqL7Ezopq+nxRFQV0ZVS2wQhQf7NOzwIll2N//HVWD74aUcfMCijB8UBy8GzeJDYBMTn9 mssXAS0c4e36NoolqwB9ARdAE+g3dA4mwF9lOd6KStg0UlMy1Gi7w2g14uIx27Njlyk4Q++SGt8ol NWZKkKIA==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003Vo-AB; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 27/51] iomap: Change calling convention for zeroing Date: Wed, 10 Jun 2020 13:13:21 -0700 Message-Id: <20200610201345.13273-28-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E327B18086E22 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Pass the full length to iomap_zero() and dax_iomap_zero(), then have them return how many bytes they actually handled. This is prepatory work for handling THP, although it looks like DAX could actually take advantage of it if there's a larger contiguous area. Signed-off-by: Matthew Wilcox (Oracle) --- fs/dax.c | 13 ++++++------- fs/iomap/buffered-io.c | 33 +++++++++++++++------------------ include/linux/dax.h | 3 +-- 3 files changed, 22 insertions(+), 27 deletions(-) diff --git a/fs/dax.c b/fs/dax.c index 11b16729b86f..18e9b5f0dedf 100644 --- a/fs/dax.c +++ b/fs/dax.c @@ -1038,18 +1038,18 @@ static vm_fault_t dax_load_hole(struct xa_state *xas, return ret; } -int dax_iomap_zero(loff_t pos, unsigned offset, unsigned size, - struct iomap *iomap) +ssize_t dax_iomap_zero(loff_t pos, loff_t length, struct iomap *iomap) { sector_t sector = iomap_sector(iomap, pos & PAGE_MASK); pgoff_t pgoff; long rc, id; void *kaddr; bool page_aligned = false; - + unsigned offset = offset_in_page(pos); + unsigned size = min_t(loff_t, PAGE_SIZE - offset, length); if (IS_ALIGNED(sector << SECTOR_SHIFT, PAGE_SIZE) && - IS_ALIGNED(size, PAGE_SIZE)) + (size == PAGE_SIZE)) page_aligned = true; rc = bdev_dax_pgoff(iomap->bdev, sector, PAGE_SIZE, &pgoff); @@ -1059,8 +1059,7 @@ int dax_iomap_zero(loff_t pos, unsigned offset, unsigned size, id = dax_read_lock(); if (page_aligned) - rc = dax_zero_page_range(iomap->dax_dev, pgoff, - size >> PAGE_SHIFT); + rc = dax_zero_page_range(iomap->dax_dev, pgoff, 1); else rc = dax_direct_access(iomap->dax_dev, pgoff, 1, &kaddr, NULL); if (rc < 0) { @@ -1073,7 +1072,7 @@ int dax_iomap_zero(loff_t pos, unsigned offset, unsigned size, dax_flush(iomap->dax_dev, kaddr + offset, size); } dax_read_unlock(id); - return 0; + return size; } static loff_t diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index fffdf5906e99..8d690ad68657 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -928,11 +928,13 @@ iomap_file_unshare(struct inode *inode, loff_t pos, loff_t len, } EXPORT_SYMBOL_GPL(iomap_file_unshare); -static int iomap_zero(struct inode *inode, loff_t pos, unsigned offset, - unsigned bytes, struct iomap *iomap, struct iomap *srcmap) +static ssize_t iomap_zero(struct inode *inode, loff_t pos, loff_t length, + struct iomap *iomap, struct iomap *srcmap) { struct page *page; int status; + unsigned offset = offset_in_page(pos); + unsigned bytes = min_t(loff_t, PAGE_SIZE - offset, length); status = iomap_write_begin(inode, pos, bytes, 0, &page, iomap, srcmap); if (status) @@ -944,38 +946,33 @@ static int iomap_zero(struct inode *inode, loff_t pos, unsigned offset, return iomap_write_end(inode, pos, bytes, bytes, page, iomap, srcmap); } -static loff_t -iomap_zero_range_actor(struct inode *inode, loff_t pos, loff_t count, - void *data, struct iomap *iomap, struct iomap *srcmap) +static loff_t iomap_zero_range_actor(struct inode *inode, loff_t pos, + loff_t length, void *data, struct iomap *iomap, + struct iomap *srcmap) { bool *did_zero = data; loff_t written = 0; - int status; /* already zeroed? we're done. */ if (srcmap->type == IOMAP_HOLE || srcmap->type == IOMAP_UNWRITTEN) - return count; + return length; do { - unsigned offset, bytes; - - offset = offset_in_page(pos); - bytes = min_t(loff_t, PAGE_SIZE - offset, count); + ssize_t bytes; if (IS_DAX(inode)) - status = dax_iomap_zero(pos, offset, bytes, iomap); + bytes = dax_iomap_zero(pos, length, iomap); else - status = iomap_zero(inode, pos, offset, bytes, iomap, - srcmap); - if (status < 0) - return status; + bytes = iomap_zero(inode, pos, length, iomap, srcmap); + if (bytes < 0) + return bytes; pos += bytes; - count -= bytes; + length -= bytes; written += bytes; if (did_zero) *did_zero = true; - } while (count > 0); + } while (length > 0); return written; } diff --git a/include/linux/dax.h b/include/linux/dax.h index 6904d4e0b2e0..0ba618aeec98 100644 --- a/include/linux/dax.h +++ b/include/linux/dax.h @@ -214,8 +214,7 @@ vm_fault_t dax_finish_sync_fault(struct vm_fault *vmf, int dax_delete_mapping_entry(struct address_space *mapping, pgoff_t index); int dax_invalidate_mapping_entry_sync(struct address_space *mapping, pgoff_t index); -int dax_iomap_zero(loff_t pos, unsigned offset, unsigned size, - struct iomap *iomap); +ssize_t dax_iomap_zero(loff_t pos, loff_t length, struct iomap *iomap); static inline bool dax_mapping(struct address_space *mapping) { return mapping->host && IS_DAX(mapping->host); From patchwork Wed Jun 10 20:13:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598831 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E7FF3618 for ; Wed, 10 Jun 2020 20:39:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B42FC206A4 for ; Wed, 10 Jun 2020 20:39:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="OiAighL9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B42FC206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C51318D0023; Wed, 10 Jun 2020 16:39:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B86198D000E; Wed, 10 Jun 2020 16:39:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A2C9B8D0023; Wed, 10 Jun 2020 16:39:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id 8BE108D000E for ; Wed, 10 Jun 2020 16:39:21 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 4F9B514767B for ; Wed, 10 Jun 2020 20:39:21 +0000 (UTC) X-FDA: 76914467322.02.move16_4f11cd826dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 2C59B147677 for ; Wed, 10 Jun 2020 20:39:21 +0000 (UTC) X-Spam-Summary: 2,0,0,44277c7e7c1c3cc9,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:2:41:69:355:379:421:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2393:2553:2559:2562:2693:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3872:3874:4118:4321:5007:6119:6261:6653:7576:7875:7903:8660:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12555:12683:12895:12986:13148:13149:13230:13894:14096:14394:21080:21212:21451:21627:21796:21939:21990:30034:30036:30046:30054:30070:30079:30090,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: move16_4f11cd826dce X-Filterd-Recvd-Size: 7309 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=C9KNZLFkt8Tle4VaX/QjFZQwjCdLqmDjaJSbY4g1BBc=; b=OiAighL9mBrPMhnrkS/gqGty1F u+yNVja/T7n3ktY+YB0D0qaJCPfbXCZl+5ippi9ekL75UScg7fDbqZKZZfN8RISA5FApVLRtZbdg5 p0jd/PBsujMOChVBoe5Yno51GObGHDl00288vESYdf5iSJKEiH7iyWk5zCgBFto9+w9teXyQzL6hs eRHuA3fYJSvvczre/GDjn6fTWi0+rF5f1F0zwZFOTJtDR1b3TqOi3Glh1clzvypR6h2AZWUBEmf6J VqYU60h2wqrw0yz/hoFGCNB2yzIv9RcVmZBNol0fiYqT6bP6Vi7e0weWLFJWRUPFOBzhXVQ0OCLel hdP4QG6w==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003Vv-BK; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 28/51] iomap: Change iomap_write_begin calling convention Date: Wed, 10 Jun 2020 13:13:22 -0700 Message-Id: <20200610201345.13273-29-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2C59B147677 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Pass (up to) the remaining length of the extent to iomap_write_begin() and have it return the number of bytes that will fit in the page. That lets us copy more bytes per call to iomap_write_begin() if the page cache has already allocated a THP (and will in future allow us to pass a hint to the page cache that it should try to allocate a larger page if there are none in the cache). Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 63 +++++++++++++++++++++++------------------- 1 file changed, 34 insertions(+), 29 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 8d690ad68657..e445ee5f0521 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -571,14 +571,14 @@ iomap_read_page_sync(loff_t block_start, struct page *page, unsigned poff, return submit_bio_wait(&bio); } -static int -__iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, int flags, - struct page *page, struct iomap *srcmap) +static ssize_t __iomap_write_begin(struct inode *inode, loff_t pos, + size_t len, int flags, struct page *page, struct iomap *srcmap) { loff_t block_size = i_blocksize(inode); loff_t block_start = pos & ~(block_size - 1); loff_t block_end = (pos + len + block_size - 1) & ~(block_size - 1); - unsigned from = offset_in_page(pos), to = from + len; + size_t from = offset_in_thp(page, pos); + size_t to = from + len; size_t poff, plen; int status; @@ -614,12 +614,13 @@ __iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, int flags, return 0; } -static int -iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, unsigned flags, - struct page **pagep, struct iomap *iomap, struct iomap *srcmap) +static ssize_t iomap_write_begin(struct inode *inode, loff_t pos, loff_t len, + unsigned flags, struct page **pagep, struct iomap *iomap, + struct iomap *srcmap) { const struct iomap_page_ops *page_ops = iomap->page_ops; struct page *page; + size_t offset; int status = 0; BUG_ON(pos + len > iomap->offset + iomap->length); @@ -630,6 +631,8 @@ iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, unsigned flags, return -EINTR; if (page_ops && page_ops->page_prepare) { + if (len > UINT_MAX) + len = UINT_MAX; status = page_ops->page_prepare(inode, pos, len, iomap); if (status) return status; @@ -641,6 +644,10 @@ iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, unsigned flags, status = -ENOMEM; goto out_no_page; } + page = thp_head(page); + offset = offset_in_thp(page, pos); + if (len > thp_size(page) - offset) + len = thp_size(page) - offset; if (srcmap->type == IOMAP_INLINE) iomap_read_inline_data(inode, page, srcmap); @@ -650,11 +657,11 @@ iomap_write_begin(struct inode *inode, loff_t pos, unsigned len, unsigned flags, status = __iomap_write_begin(inode, pos, len, flags, page, srcmap); - if (unlikely(status)) + if (status < 0) goto out_unlock; *pagep = page; - return 0; + return len; out_unlock: unlock_page(page); @@ -809,8 +816,10 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data, status = iomap_write_begin(inode, pos, bytes, 0, &page, iomap, srcmap); - if (unlikely(status)) + if (status < 0) break; + /* We may be partway through a THP */ + offset = offset_in_thp(page, pos); if (mapping_writably_mapped(inode->i_mapping)) flush_dcache_page(page); @@ -872,8 +881,7 @@ static loff_t iomap_unshare_actor(struct inode *inode, loff_t pos, loff_t length, void *data, struct iomap *iomap, struct iomap *srcmap) { - long status = 0; - ssize_t written = 0; + loff_t written = 0; /* don't bother with blocks that are not shared to start with */ if (!(iomap->flags & IOMAP_F_SHARED)) @@ -883,25 +891,24 @@ iomap_unshare_actor(struct inode *inode, loff_t pos, loff_t length, void *data, return length; do { - unsigned long offset = offset_in_page(pos); - unsigned long bytes = min_t(loff_t, PAGE_SIZE - offset, length); struct page *page; + ssize_t bytes; - status = iomap_write_begin(inode, pos, bytes, + bytes = iomap_write_begin(inode, pos, length, IOMAP_WRITE_F_UNSHARE, &page, iomap, srcmap); - if (unlikely(status)) - return status; + if (bytes < 0) + return bytes; - status = iomap_write_end(inode, pos, bytes, bytes, page, iomap, + bytes = iomap_write_end(inode, pos, bytes, bytes, page, iomap, srcmap); - if (WARN_ON_ONCE(status == 0)) + if (WARN_ON_ONCE(bytes == 0)) return -EIO; cond_resched(); - pos += status; - written += status; - length -= status; + pos += bytes; + written += bytes; + length -= bytes; balance_dirty_pages_ratelimited(inode->i_mapping); } while (length); @@ -932,15 +939,13 @@ static ssize_t iomap_zero(struct inode *inode, loff_t pos, loff_t length, struct iomap *iomap, struct iomap *srcmap) { struct page *page; - int status; - unsigned offset = offset_in_page(pos); - unsigned bytes = min_t(loff_t, PAGE_SIZE - offset, length); + ssize_t bytes; - status = iomap_write_begin(inode, pos, bytes, 0, &page, iomap, srcmap); - if (status) - return status; + bytes = iomap_write_begin(inode, pos, length, 0, &page, iomap, srcmap); + if (bytes < 0) + return bytes; - zero_user(page, offset, bytes); + zero_user(page, offset_in_thp(page, pos), bytes); mark_page_accessed(page); return iomap_write_end(inode, pos, bytes, bytes, page, iomap, srcmap); From patchwork Wed Jun 10 20:13:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598827 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1B528618 for ; Wed, 10 Jun 2020 20:39:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CF48D206A4 for ; Wed, 10 Jun 2020 20:39:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="abB0wGDo" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CF48D206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 87E5A8D0021; Wed, 10 Jun 2020 16:39:17 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 80A1D8D000E; Wed, 10 Jun 2020 16:39:17 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 680748D0021; Wed, 10 Jun 2020 16:39:17 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0106.hostedemail.com [216.40.44.106]) by kanga.kvack.org (Postfix) with ESMTP id 50C958D000E for ; Wed, 10 Jun 2020 16:39:17 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 18C62146F59 for ; Wed, 10 Jun 2020 20:39:17 +0000 (UTC) X-FDA: 76914467154.13.war20_060dd9f26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id E06231813F3F1 for ; Wed, 10 Jun 2020 20:39:16 +0000 (UTC) X-Spam-Summary: 2,0,0,304a59cddec87d5e,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:2:41:69:355:371:372:379:421:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:2198:2199:2393:2559:2562:2898:3138:3139:3140:3141:3142:3308:3865:3866:3867:3868:3870:3871:3872:3874:4049:4118:4250:4321:4605:5007:6119:6261:6653:7576:7875:7901:8603:8957:9036:10004:10226:11026:11473:11658:11914:12043:12291:12296:12297:12438:12555:12683:12895:12986:13894:14096:14110:14394:21080:21324:21451:21627:21795:21990:30046:30051:30054:30070:30079,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: war20_060dd9f26dce X-Filterd-Recvd-Size: 7864 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=X3jPEAbYsbcdPk9lVXyvLsl+/O0MuFQGp8os2fJmDlE=; b=abB0wGDoYqPJf3a0kTiTdIqZhw O8Z2N2CbGR7yjgwE6PI/y1fk4DQuQ93VOyG1Cl++6rTIE8YVeRKpDQUq6BZnZ3ngX1lNMXCQ39pX5 htRAiPu9klOmqT/TyhlPSdfPHOvQ0gSaAC+MVfdzn207YMHp7m8D0Ij0qSYbxIr6tt+AUO9HNm46G a/VgbkdZ43HCD8fDGJeGLj4unnZFe0N4fMzMzHxrx5yunT2P+IAhsNf+qKdMwqiJl9Dic1sFL4hF+ n6H2DOGftvQjd7p8lippSF18uqHWUZAF+n/iiasPoIrTkfuI3uZ1eXFuiO4dbFoyi2i+OT7+wxz/H yHMBC1Dg==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003W2-Dl; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 29/51] iomap: Support THPs in write paths Date: Wed, 10 Jun 2020 13:13:23 -0700 Message-Id: <20200610201345.13273-30-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E06231813F3F1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Use thp_size() instead of PAGE_SIZE and offset_in_thp() instead of offset_in_page(). Also simplify the logic in iomap_do_writepage() for determining end of file. Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 44 +++++++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index e445ee5f0521..9275268ea97e 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -460,7 +460,7 @@ iomap_is_partially_uptodate(struct page *page, unsigned long from, unsigned i; /* Limit range to one page */ - len = min_t(unsigned, PAGE_SIZE - from, count); + len = min_t(unsigned, thp_size(page) - from, count); /* First and last blocks in range within page */ first = from >> inode->i_blkbits; @@ -654,8 +654,8 @@ static ssize_t iomap_write_begin(struct inode *inode, loff_t pos, loff_t len, else if (iomap->flags & IOMAP_F_BUFFER_HEAD) status = __block_write_begin_int(page, pos, len, NULL, srcmap); else - status = __iomap_write_begin(inode, pos, len, flags, page, - srcmap); + status = __iomap_write_begin(inode, pos, len, flags, + thp_head(page), srcmap); if (status < 0) goto out_unlock; @@ -717,7 +717,7 @@ static size_t __iomap_write_end(struct inode *inode, loff_t pos, size_t len, */ if (unlikely(copied < len && !PageUptodate(page))) return 0; - iomap_set_range_uptodate(page, offset_in_page(pos), len); + iomap_set_range_uptodate(page, offset_in_thp(page, pos), len); iomap_set_page_dirty(page); return copied; } @@ -753,7 +753,8 @@ static size_t iomap_write_end(struct inode *inode, loff_t pos, size_t len, ret = block_write_end(NULL, inode->i_mapping, pos, len, copied, page, NULL); } else { - ret = __iomap_write_end(inode, pos, len, copied, page); + ret = __iomap_write_end(inode, pos, len, copied, + thp_head(page)); } /* @@ -792,6 +793,10 @@ iomap_write_actor(struct inode *inode, loff_t pos, loff_t length, void *data, unsigned long bytes; /* Bytes to write to page */ size_t copied; /* Bytes copied from user */ + /* + * XXX: We don't know what size page we'll find in the + * page cache, so only copy up to a regular page boundary. + */ offset = offset_in_page(pos); bytes = min_t(unsigned long, PAGE_SIZE - offset, iov_iter_count(i)); @@ -1116,7 +1121,7 @@ iomap_finish_ioend(struct iomap_ioend *ioend, int error) next = bio->bi_private; /* walk each page on bio, ending page IO on them */ - bio_for_each_segment_all(bv, bio, iter_all) + bio_for_each_thp_segment_all(bv, bio, iter_all) iomap_finish_page_writeback(inode, bv->bv_page, error); bio_put(bio); } @@ -1322,7 +1327,7 @@ iomap_add_to_ioend(struct inode *inode, loff_t offset, struct page *page, { sector_t sector = iomap_sector(&wpc->iomap, offset); unsigned len = i_blocksize(inode); - unsigned poff = offset & (PAGE_SIZE - 1); + unsigned poff = offset & (thp_size(page) - 1); bool merged, same_page = false; if (!wpc->ioend || !iomap_can_add_to_ioend(wpc, offset, sector)) { @@ -1372,8 +1377,9 @@ iomap_writepage_map(struct iomap_writepage_ctx *wpc, struct iomap_page *iop = iomap_page_create(inode, page); struct iomap_ioend *ioend, *next; unsigned len = i_blocksize(inode); - u64 file_offset; /* file offset of page */ + loff_t pos; int error = 0, count = 0, i; + int nr_blocks = i_blocks_per_page(inode, page); LIST_HEAD(submit_list); WARN_ON_ONCE(iop && atomic_read(&iop->write_count) != 0); @@ -1383,20 +1389,20 @@ iomap_writepage_map(struct iomap_writepage_ctx *wpc, * end of the current map or find the current map invalid, grab a new * one. */ - for (i = 0, file_offset = page_offset(page); - i < (PAGE_SIZE >> inode->i_blkbits) && file_offset < end_offset; - i++, file_offset += len) { + for (i = 0, pos = page_offset(page); + i < nr_blocks && pos < end_offset; + i++, pos += len) { if (iop && !test_bit(i, iop->uptodate)) continue; - error = wpc->ops->map_blocks(wpc, inode, file_offset); + error = wpc->ops->map_blocks(wpc, inode, pos); if (error) break; if (WARN_ON_ONCE(wpc->iomap.type == IOMAP_INLINE)) continue; if (wpc->iomap.type == IOMAP_HOLE) continue; - iomap_add_to_ioend(inode, file_offset, page, iop, wpc, wbc, + iomap_add_to_ioend(inode, pos, page, iop, wpc, wbc, &submit_list); count++; } @@ -1478,7 +1484,6 @@ iomap_do_writepage(struct page *page, struct writeback_control *wbc, void *data) { struct iomap_writepage_ctx *wpc = data; struct inode *inode = page->mapping->host; - pgoff_t end_index; u64 end_offset; loff_t offset; @@ -1519,10 +1524,8 @@ iomap_do_writepage(struct page *page, struct writeback_control *wbc, void *data) * ---------------------------------^------------------| */ offset = i_size_read(inode); - end_index = offset >> PAGE_SHIFT; - if (page->index < end_index) - end_offset = (loff_t)(page->index + 1) << PAGE_SHIFT; - else { + end_offset = page_offset(page) + thp_size(page); + if (end_offset > offset) { /* * Check whether the page to write out is beyond or straddles * i_size or not. @@ -1534,7 +1537,8 @@ iomap_do_writepage(struct page *page, struct writeback_control *wbc, void *data) * | | Straddles | * ---------------------------------^-----------|--------| */ - unsigned offset_into_page = offset & (PAGE_SIZE - 1); + unsigned offset_into_page = offset_in_thp(page, offset); + pgoff_t end_index = offset >> PAGE_SHIFT; /* * Skip the page if it is fully outside i_size, e.g. due to a @@ -1565,7 +1569,7 @@ iomap_do_writepage(struct page *page, struct writeback_control *wbc, void *data) * memory is zeroed when mapped, and writes to that region are * not written out to the file." */ - zero_user_segment(page, offset_into_page, PAGE_SIZE); + zero_user_segment(page, offset_into_page, thp_size(page)); /* Adjust the end_offset to the end of file */ end_offset = offset; From patchwork Wed Jun 10 20:13:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598859 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C749618 for ; Wed, 10 Jun 2020 20:39:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ED7BC206A4 for ; Wed, 10 Jun 2020 20:39:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="F4zTFIQB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED7BC206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B8EB58D0030; Wed, 10 Jun 2020 16:39:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AF49C8D002D; Wed, 10 Jun 2020 16:39:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 882058D0030; Wed, 10 Jun 2020 16:39:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0082.hostedemail.com [216.40.44.82]) by kanga.kvack.org (Postfix) with ESMTP id 5BCD28D002D for ; Wed, 10 Jun 2020 16:39:32 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 231D88019DD7 for ; Wed, 10 Jun 2020 20:39:32 +0000 (UTC) X-FDA: 76914467784.16.sign81_00151bc26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id E828010272E24 for ; Wed, 10 Jun 2020 20:39:31 +0000 (UTC) X-Spam-Summary: 2,0,0,e7b829d78773c735,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1714:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3351:3865:3870:3874:4321:5007:6117:6261:6653:7576:7875:8603:8957:10004:11026:11473:11658:11914:12043:12297:12438:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21451:21627:21990:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: sign81_00151bc26dce X-Filterd-Recvd-Size: 2688 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=33n6uHIDU/xwgJxMoeJs++K6GI7dTUyUJxhDT49yQow=; b=F4zTFIQBmlQAqxKjoc16EhcY4g VDaIF7Tnc7ZuUPpoYpIo4Hs+R4wqGUFkQFSMA5eilS6i+6eHGnqeaZqDu/0FcdTIY4pvRjcJURHeZ hxQ4WaKXHesAjK4W6g7UOoRUHkeVt/mQr1tOQGEqz2tTyHYMcVYczOnssY+QZkP9XFIK/8gfm0SSY 0mJFXwxZA4noGIpK3rBov1HwCkm8HCsZw+8SKbXOpAdVUrb5fOtzQxE2v15zudkDTyySSAveZ4QGF KBBADkbLpfr8h+l4/EUjBTxJKKZoMDc4gXlhrO/i12MaLUXcUiHZwK4MxzISo3xA4SzV/KFSJWIxS XLKP1Cpw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003W9-F4; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Hellwig Subject: [PATCH v6 30/51] iomap: Inline data shouldn't see THPs Date: Wed, 10 Jun 2020 13:13:24 -0700 Message-Id: <20200610201345.13273-31-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: E828010272E24 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Assert that we're not seeing THPs in functions that read/write inline data, rather than zeroing out the tail. Signed-off-by: Matthew Wilcox (Oracle) Reviewed-by: Christoph Hellwig --- fs/iomap/buffered-io.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 9275268ea97e..318c1ecc18c0 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -215,6 +215,7 @@ iomap_read_inline_data(struct inode *inode, struct page *page, return; BUG_ON(page->index); + BUG_ON(PageCompound(page)); BUG_ON(size > PAGE_SIZE - offset_in_page(iomap->inline_data)); addr = kmap_atomic(page); @@ -728,6 +729,7 @@ static size_t iomap_write_end_inline(struct inode *inode, struct page *page, void *addr; WARN_ON_ONCE(!PageUptodate(page)); + BUG_ON(PageCompound(page)); BUG_ON(pos + copied > PAGE_SIZE - offset_in_page(iomap->inline_data)); addr = kmap_atomic(page); From patchwork Wed Jun 10 20:13:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598829 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E7F4890 for ; Wed, 10 Jun 2020 20:39:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B4E81206A4 for ; Wed, 10 Jun 2020 20:39:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="igTBtfUm" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B4E81206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B8B048D0022; Wed, 10 Jun 2020 16:39:20 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AEA808D000E; Wed, 10 Jun 2020 16:39:20 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F0028D0022; Wed, 10 Jun 2020 16:39:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0197.hostedemail.com [216.40.44.197]) by kanga.kvack.org (Postfix) with ESMTP id 741CD8D000E for ; Wed, 10 Jun 2020 16:39:20 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 33B511803545F for ; Wed, 10 Jun 2020 20:39:20 +0000 (UTC) X-FDA: 76914467280.23.kick89_4d081cc26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 039DB19F77 for ; Wed, 10 Jun 2020 20:39:19 +0000 (UTC) X-Spam-Summary: 2,0,0,b9702688c3559a3f,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1540:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3868:3871:4321:5007:6261:6653:7576:10004:11026:11658:11914:12043:12296:12297:12438:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21451:21627:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: kick89_4d081cc26dce X-Filterd-Recvd-Size: 2537 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=NF4IbJ7tnb1BDecklJKAGawEVJngscEaY/hk+Y65vZs=; b=igTBtfUmXt8Mznyh9YOcO59toO 3vqpDxaK4c26kTmcie0ib46b+joeV0vZI044ZEzVKnl9axARAjdnM5utCYzbPKHgKBNio/S2phuVn 2ucEZ+AlaK7ZFrwiwb6G8bnHHA/6Y3Sck4HLUVAX4Piv1COkFXSTQG6BFlnNIIHNQFKdLXlj4G606 jxBeKSdAi86B4Yxpk2/HYIhrKw23tZsWvZvfjsvKZOBJ+SRX1VtQ8+auhYf853jKeZ+weKscFrH5C ZRn3S3st12cQtogLDrb3CGF56kritIFeUu6l5naRJaSbRf1kWwDnehV03zRwIeaFvMdNdEwQpDkUV 9sE8Iz4Q==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003WG-Ga; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 31/51] iomap: Handle tail pages in iomap_page_mkwrite Date: Wed, 10 Jun 2020 13:13:25 -0700 Message-Id: <20200610201345.13273-32-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 039DB19F77 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" iomap_page_mkwrite() can be called with a tail page. If we are, operate on the head page, since we're treating the entire thing as a single unit and the whole page is dirtied at the same time. Signed-off-by: Matthew Wilcox (Oracle) --- fs/iomap/buffered-io.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 318c1ecc18c0..8359e369c4b5 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -1046,7 +1046,7 @@ iomap_page_mkwrite_actor(struct inode *inode, loff_t pos, loff_t length, vm_fault_t iomap_page_mkwrite(struct vm_fault *vmf, const struct iomap_ops *ops) { - struct page *page = vmf->page; + struct page *page = thp_head(vmf->page); struct inode *inode = file_inode(vmf->vma->vm_file); unsigned long length; loff_t offset; From patchwork Wed Jun 10 20:13:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598853 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4DD0A90 for ; Wed, 10 Jun 2020 20:39:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 10981206A4 for ; Wed, 10 Jun 2020 20:39:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="avLEpxx7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 10981206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D10F68D002C; Wed, 10 Jun 2020 16:39:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C79D38D002D; Wed, 10 Jun 2020 16:39:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 91EED8D002C; Wed, 10 Jun 2020 16:39:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 52BC18D002C for ; Wed, 10 Jun 2020 16:39:30 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 18FA8140482 for ; Wed, 10 Jun 2020 20:39:30 +0000 (UTC) X-FDA: 76914467700.10.cloud79_210c3fd26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id D58384403 for ; Wed, 10 Jun 2020 20:39:29 +0000 (UTC) X-Spam-Summary: 2,0,0,db5af33112ea5c23,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:3871:3872:4321:4384:5007:6261:6653:7576:8603:10004:11026:11473:11658:11914:12297:12438:12555:12895:12986:13069:13161:13229:13311:13357:13894:14096:14181:14384:14394:14721:21080:21433:21451:21627:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: cloud79_210c3fd26dce X-Filterd-Recvd-Size: 2799 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=U0445aIDxVE7s5h/BfzJEUrieaS8YnImXhJbP+y1GVw=; b=avLEpxx7/Y85V7Bgil5FWjSuUz Y9Br6oIui4chclLNI97QBmGVpP5tL0Ahw9/sK40R9/cZ+b0N/mLYtaRo44ioLOo7LJqITdgcIW0sM ydYglPZknDTili3m0Pn82JN87Qah17ytDgQlYACs/YIpFJK2UXORJJmsvkcPZpL8jVA/5PpOaB2Ns WFsDWcrZHde1uYG+Oh10JPuQ80bmbEth8a9xXyNdcFMqBRMNPZrpSoQ/hi235Qeife1KO/2d4WSxv vlW/4W/Y9Y5IzIHuhZ9y8AArbskwFolKJu6St3Ki3T/PPjLjH5IePLXYjSs1wm8h/Gv+RJZVjm59J si12XtKw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003WN-Hn; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 32/51] xfs: Support THPs Date: Wed, 10 Jun 2020 13:13:26 -0700 Message-Id: <20200610201345.13273-33-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D58384403 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" There is one place which assumes the size of a page; fix it. Signed-off-by: Matthew Wilcox (Oracle) --- fs/xfs/xfs_aops.c | 2 +- fs/xfs/xfs_super.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 55d126d4e096..20968842b2f0 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -548,7 +548,7 @@ xfs_discard_page( if (error && !XFS_FORCED_SHUTDOWN(mp)) xfs_alert(mp, "page discard unable to remove delalloc mapping."); out_invalidate: - iomap_invalidatepage(page, 0, PAGE_SIZE); + iomap_invalidatepage(page, 0, thp_size(page)); } static const struct iomap_writeback_ops xfs_writeback_ops = { diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c index 379cbff438bc..1a4a7a8766db 100644 --- a/fs/xfs/xfs_super.c +++ b/fs/xfs/xfs_super.c @@ -1829,7 +1829,7 @@ static struct file_system_type xfs_fs_type = { .init_fs_context = xfs_init_fs_context, .parameters = xfs_fs_parameters, .kill_sb = kill_block_super, - .fs_flags = FS_REQUIRES_DEV, + .fs_flags = FS_REQUIRES_DEV | FS_THP_SUPPORT, }; MODULE_ALIAS_FS("xfs"); From patchwork Wed Jun 10 20:13:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598841 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D348618 for ; Wed, 10 Jun 2020 20:39:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 600A7206A4 for ; Wed, 10 Jun 2020 20:39:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="EZw153FE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 600A7206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C17258D000E; Wed, 10 Jun 2020 16:39:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B8B358D0029; Wed, 10 Jun 2020 16:39:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 991538D000E; Wed, 10 Jun 2020 16:39:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0179.hostedemail.com [216.40.44.179]) by kanga.kvack.org (Postfix) with ESMTP id 6E8898D0027 for ; Wed, 10 Jun 2020 16:39:26 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2E5A6180AEF50 for ; Wed, 10 Jun 2020 20:39:26 +0000 (UTC) X-FDA: 76914467532.01.price65_1715e0e26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 030C218002068 for ; Wed, 10 Jun 2020 20:39:25 +0000 (UTC) X-Spam-Summary: 2,0,0,249422be182acc44,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2731:2914:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3870:3874:4250:4321:4385:4605:5007:6119:6261:6653:7576:10004:10226:11026:11473:11658:11914:12043:12296:12297:12438:12555:12895:12986:13894:14096:14181:14394:14721:21080:21451:21627:21990:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: price65_1715e0e26dce X-Filterd-Recvd-Size: 3991 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=luSvxdXsOgCclUvjQm/qpAoORhRv4yOA5W917enf22U=; b=EZw153FEH9cU4q+22LkbZ0J1/e pchuuw1pyd8pHVSUTDS9Sls/AXzdYOAfz06LgP6YdOEAfZ5FxBVeMEvKzJZ73kwTTnG5Ci74HQxNW bL0dWOxFRrECSWsTQOI5vRlouVRA/1wsPZJZSeud2EiFwkBr7lv7E5WrcGTlAt7AcWY06yUZmTODM PN2g5WHGui2lgwfxiVhL9KoY2FvbiVHwAurqYffFTbcaoPiGrLKDl/EBbuG6jQGCm59KBEGBKBJEf M/Z6nXRUhn5QggBa25gjAe1WRbmKGz/oLKc6U7R/cMVzCMdJpzneM/ztfl7CqBKlwVTojgsS9cNTz WWiEdG9A==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003WV-JI; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A . Shutemov" Subject: [PATCH v6 33/51] mm: Make prep_transhuge_page return its argument Date: Wed, 10 Jun 2020 13:13:27 -0700 Message-Id: <20200610201345.13273-34-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 030C218002068 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" By permitting NULL or order-0 pages as an argument, and returning the argument, callers can write: return prep_transhuge_page(alloc_pages(...)); instead of assigning the result to a temporary variable and conditionally passing that to prep_transhuge_page(). Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Kirill A. Shutemov --- include/linux/huge_mm.h | 7 +++++-- mm/huge_memory.c | 9 +++++++-- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index d125912a3e0d..61de7e8683cd 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -193,7 +193,7 @@ extern unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long len, unsigned long pgoff, unsigned long flags); -extern void prep_transhuge_page(struct page *page); +extern struct page *prep_transhuge_page(struct page *page); extern void free_transhuge_page(struct page *page); bool is_transparent_hugepage(struct page *page); @@ -381,7 +381,10 @@ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma, return false; } -static inline void prep_transhuge_page(struct page *page) {} +static inline struct page *prep_transhuge_page(struct page *page) +{ + return page; +} static inline bool is_transparent_hugepage(struct page *page) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 78c84bee7e29..80fb38e93837 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -508,15 +508,20 @@ static inline struct deferred_split *get_deferred_split_queue(struct page *page) } #endif -void prep_transhuge_page(struct page *page) +struct page *prep_transhuge_page(struct page *page) { + if (!page || compound_order(page) == 0) + return page; /* - * we use page->mapping and page->indexlru in second tail page + * we use page->mapping and page->index in second tail page * as list_head: assuming THP order >= 2 */ + BUG_ON(compound_order(page) == 1); INIT_LIST_HEAD(page_deferred_list(page)); set_compound_page_dtor(page, TRANSHUGE_PAGE_DTOR); + + return page; } bool is_transparent_hugepage(struct page *page) From patchwork Wed Jun 10 20:13:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598863 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ACB48618 for ; Wed, 10 Jun 2020 20:39:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7940E20825 for ; Wed, 10 Jun 2020 20:39:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="fZ2qNCLZ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7940E20825 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 397E18D0034; Wed, 10 Jun 2020 16:39:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3489F8D002D; Wed, 10 Jun 2020 16:39:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A1D18D0034; Wed, 10 Jun 2020 16:39:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0018.hostedemail.com [216.40.44.18]) by kanga.kvack.org (Postfix) with ESMTP id EB7F58D0033 for ; Wed, 10 Jun 2020 16:39:33 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A7D8A1801C377 for ; Wed, 10 Jun 2020 20:39:33 +0000 (UTC) X-FDA: 76914467826.01.shock40_170a27826dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 7E4831801736E for ; Wed, 10 Jun 2020 20:39:33 +0000 (UTC) X-Spam-Summary: 2,0,0,a846be791124f103,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:560:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3353:3867:3874:4321:4605:5007:6119:6261:6653:7576:10004:11026:11473:11658:11914:12043:12114:12291:12296:12297:12438:12555:12895:12986:13894:14096:14181:14394:14721:21080:21451:21627:21990:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: shock40_170a27826dce X-Filterd-Recvd-Size: 4401 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf44.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=GYkASM4BgicRB6qQw/ipKWc7P4OsD4qqbRXZJoBJj2A=; b=fZ2qNCLZY8A3ORQkyJtDxInaEG Ecup0o0tQdAskbPlcKAGJkERx1rVwDVpliXE0v56sOxSjvNkNJfecQ3fVI58HSMbEXzGNKOdBif3G DXv8jO2AifgU8NIxn1NHR75q907XSarxq4GYn0ljEUeXXRRH2uVyC9ouBMtk3HWOWFhdJKjmh5XBl MlZyubcdGQ69V9DNZj48RpjKvCg3Oi69Hv2gSpQc+SUwkcQXvWTQu4CfFKgVQoMuBFxVCWufXTOsZ h/Hl+P38eVcPsrviQQgN1txwem/H/6DkT+MZrF0J7Za5Edimj9JmwlOCN09qAxpYv0TOgmM7kvrUO yp8bAmgw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003Wa-LC; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A . Shutemov" Subject: [PATCH v6 34/51] mm: Add __page_cache_alloc_order Date: Wed, 10 Jun 2020 13:13:28 -0700 Message-Id: <20200610201345.13273-35-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7E4831801736E X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" This new function allows page cache pages to be allocated that are larger than an order-0 page. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Kirill A. Shutemov --- include/linux/pagemap.h | 24 +++++++++++++++++++++--- mm/filemap.c | 12 ++++++++---- 2 files changed, 29 insertions(+), 7 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index f47ba9f18f3e..8455a3e16900 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -280,15 +280,33 @@ static inline void *detach_page_private(struct page *page) return data; } +static inline gfp_t thp_gfpmask(gfp_t gfp) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + /* We'd rather allocate smaller pages than stall a page fault */ + gfp |= GFP_TRANSHUGE_LIGHT; + gfp &= ~__GFP_DIRECT_RECLAIM; +#endif + return gfp; +} + #ifdef CONFIG_NUMA -extern struct page *__page_cache_alloc(gfp_t gfp); +extern struct page *__page_cache_alloc_order(gfp_t gfp, unsigned int order); #else -static inline struct page *__page_cache_alloc(gfp_t gfp) +static inline +struct page *__page_cache_alloc_order(gfp_t gfp, unsigned int order) { - return alloc_pages(gfp, 0); + if (order == 0) + return alloc_pages(gfp, 0); + return prep_transhuge_page(alloc_pages(thp_gfpmask(gfp), order)); } #endif +static inline struct page *__page_cache_alloc(gfp_t gfp) +{ + return __page_cache_alloc_order(gfp, 0); +} + static inline struct page *page_cache_alloc(struct address_space *x) { return __page_cache_alloc(mapping_gfp_mask(x)); diff --git a/mm/filemap.c b/mm/filemap.c index 80ce3658b147..3a9579a1ffa7 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -938,24 +938,28 @@ int add_to_page_cache_lru(struct page *page, struct address_space *mapping, EXPORT_SYMBOL_GPL(add_to_page_cache_lru); #ifdef CONFIG_NUMA -struct page *__page_cache_alloc(gfp_t gfp) +struct page *__page_cache_alloc_order(gfp_t gfp, unsigned int order) { int n; struct page *page; + if (order > 0) + gfp = thp_gfpmask(gfp); + if (cpuset_do_page_mem_spread()) { unsigned int cpuset_mems_cookie; do { cpuset_mems_cookie = read_mems_allowed_begin(); n = cpuset_mem_spread_node(); - page = __alloc_pages_node(n, gfp, 0); + page = __alloc_pages_node(n, gfp, order); + prep_transhuge_page(page); } while (!page && read_mems_allowed_retry(cpuset_mems_cookie)); return page; } - return alloc_pages(gfp, 0); + return prep_transhuge_page(alloc_pages(gfp, order)); } -EXPORT_SYMBOL(__page_cache_alloc); +EXPORT_SYMBOL(__page_cache_alloc_order); #endif /* From patchwork Wed Jun 10 20:13:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598865 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E3FF690 for ; Wed, 10 Jun 2020 20:39:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B0D67206A4 for ; Wed, 10 Jun 2020 20:39:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="I/97wCaN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B0D67206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 66FBE8D002D; Wed, 10 Jun 2020 16:39:34 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 45AE48D0035; Wed, 10 Jun 2020 16:39:34 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 237428D0033; Wed, 10 Jun 2020 16:39:34 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0252.hostedemail.com [216.40.44.252]) by kanga.kvack.org (Postfix) with ESMTP id E6EF48D002D for ; Wed, 10 Jun 2020 16:39:33 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A1EDE3EEFA for ; Wed, 10 Jun 2020 20:39:33 +0000 (UTC) X-FDA: 76914467826.27.waste84_07147b826dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id 1EC4D3DF40 for ; Wed, 10 Jun 2020 20:39:33 +0000 (UTC) X-Spam-Summary: 2,0,0,1b1c88e364e0a3b3,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3354:3865:3866:3867:3870:3871:3872:3874:4321:5007:6261:6653:7576:7901:7903:8957:9592:10004:11026:11473:11658:11914:12043:12291:12297:12438:12555:12679:12683:12895:12986:13161:13210:13229:13894:14096:14181:14394:14721:21080:21324:21433:21450:21451:21627:21990:30006:30012:30054:30070,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: waste84_07147b826dce X-Filterd-Recvd-Size: 4580 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=82Nf07arRh2ddQXM6dUk5PZd/nwdd9dRkn6QAYZMbpg=; b=I/97wCaN/5C2K/kQyPuyQujzpa YRWA5GcqLOs2vkpFgnLJWaJQB6T7QxNfsMd3eeQR3Feu818PYzH92vaasIMMZwbhHysfiLYZX132s LcfFTuSOfhquq7DCwyyFfOMqyJcjGntmxql5tDNPn9tz6JkI9mprEHH7knvvaGIDehpNUbXE9kKS6 vZlNmBOmKt57BgcUPR9dvCnEf5wCxJ1gj+jR7vnPA1C2HpBHFR5J2tP8FD4UafK3HRBsGWjeyrn7n i0Mxe2FQN7xAApt7xHyYyhGKmGeFDkWZMNErILFRbTrxvgwBJwPPfyqu9OztjzizUtMCEyL9g5e5b KZwtQDiQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003Wg-MM; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 35/51] mm: Allow THPs to be added to the page cache Date: Wed, 10 Jun 2020 13:13:29 -0700 Message-Id: <20200610201345.13273-36-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1EC4D3DF40 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" We return -EEXIST if there are any non-shadow entries in the page cache in the range covered by the THP. If there are multiple shadow entries in the range, we set *shadowp to one of them (currently the one at the highest index). If that turns out to be the wrong answer, we can implement something more complex. This is mostly modelled after the equivalent function in the shmem code. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 52 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 17 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 3a9579a1ffa7..ab9746aff766 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -834,41 +834,58 @@ static int __add_to_page_cache_locked(struct page *page, XA_STATE(xas, &mapping->i_pages, offset); int huge = PageHuge(page); int error; + unsigned int nr = 1; void *old; VM_BUG_ON_PAGE(!PageLocked(page), page); VM_BUG_ON_PAGE(PageSwapBacked(page), page); mapping_set_update(&xas, mapping); - get_page(page); - page->mapping = mapping; - page->index = offset; - if (!huge) { error = mem_cgroup_charge(page, current->mm, gfp_mask); if (error) - goto error; + return error; + xas_set_order(&xas, offset, thp_order(page)); + nr = thp_nr_pages(page); } + page_ref_add(page, nr); + page->mapping = mapping; + page->index = offset; + do { + unsigned long exceptional = 0; + unsigned int i = 0; + xas_lock_irq(&xas); - old = xas_load(&xas); - if (old && !xa_is_value(old)) - xas_set_err(&xas, -EEXIST); - xas_store(&xas, page); + xas_for_each_conflict(&xas, old) { + if (!xa_is_value(old)) { + xas_set_err(&xas, -EEXIST); + break; + } + exceptional++; + if (shadowp) + *shadowp = old; + } + xas_create_range(&xas); if (xas_error(&xas)) goto unlock; - if (xa_is_value(old)) { - mapping->nrexceptional--; - if (shadowp) - *shadowp = old; +next: + xas_store(&xas, page); + if (++i < nr) { + xas_next(&xas); + goto next; } - mapping->nrpages++; + mapping->nrexceptional -= exceptional; + mapping->nrpages += nr; /* hugetlb pages do not participate in page cache accounting */ - if (!huge) - __inc_lruvec_page_state(page, NR_FILE_PAGES); + if (!huge) { + __mod_lruvec_page_state(page, NR_FILE_PAGES, nr); + if (nr > 1) + __inc_lruvec_page_state(page, NR_FILE_THPS); + } unlock: xas_unlock_irq(&xas); } while (xas_nomem(&xas, gfp_mask & GFP_RECLAIM_MASK)); @@ -883,7 +900,8 @@ static int __add_to_page_cache_locked(struct page *page, error: page->mapping = NULL; /* Leave page->index set: truncation relies upon it */ - put_page(page); + page_ref_sub(page, nr); + VM_BUG_ON_PAGE(page_count(page) <= 0, page); return error; } ALLOW_ERROR_INJECTION(__add_to_page_cache_locked, ERRNO); From patchwork Wed Jun 10 20:13:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598833 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD8C2618 for ; Wed, 10 Jun 2020 20:39:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A0DA1206A4 for ; Wed, 10 Jun 2020 20:39:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="n3ImGHOj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A0DA1206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 300438D0024; Wed, 10 Jun 2020 16:39:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 23D3C8D000E; Wed, 10 Jun 2020 16:39:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE15A8D0024; Wed, 10 Jun 2020 16:39:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0240.hostedemail.com [216.40.44.240]) by kanga.kvack.org (Postfix) with ESMTP id D74908D000E for ; Wed, 10 Jun 2020 16:39:22 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 9899480DADA0 for ; Wed, 10 Jun 2020 20:39:22 +0000 (UTC) X-FDA: 76914467364.19.arm14_4d009e526dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 6B96E1AB9C2 for ; Wed, 10 Jun 2020 20:39:22 +0000 (UTC) X-Spam-Summary: 2,0,0,9091c21898176d73,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:966:973:988:989:1260:1311:1314:1345:1359:1515:1534:1539:1567:1711:1714:1730:1747:1777:1792:2196:2199:2393:2559:2562:3138:3139:3140:3141:3142:3874:3876:4321:4385:5007:6117:6261:6653:7576:7901:8957:10004:11026:11473:11658:11914:12043:12296:12297:12438:12555:12895:13069:13311:13357:13894:14181:14384:14394:14721:21080:21451:21627:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:15,LUA_SUMMARY:none X-HE-Tag: arm14_4d009e526dce X-Filterd-Recvd-Size: 2321 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf17.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=GNlEMLfntSDE532epxm8/a2szgwh2sOQWwXDZZAut5w=; b=n3ImGHOjuUO9zIU7NnGb9iTHx7 4sa3MibmY3FJdosztsimDOYfZBl7yLld3xeEzyDKGf83WLEqWZVZJxZu2jmNx8zRWYx9KhGV7wbbF GG1x0oC9vXRPN2ZaoPXK8/UEnBq2vRmLPg1Do8sL33L6iw+BCGAL5T3kwxZ+FVjMmoLWIjwwzUqL6 jKLdsn4yqyECBldoeveUcK/HHHx8DYtHYp0GVmX27vajxTxTGs/ODW+C8yNUl1wX7zQFpU8a/Bved ZENUz3vpikMfHQ4JuWtOClDmFfmnMUOoVzt3nUHS4qrhnrfp6HLTRIGDX6QzIbPyVxkQ+B42qF1ke HkEiZzhw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003Wn-Nc; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 36/51] mm: Allow THPs to be removed from the page cache Date: Wed, 10 Jun 2020 13:13:30 -0700 Message-Id: <20200610201345.13273-37-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6B96E1AB9C2 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" page_cache_free_page() assumes THPs are PMD_SIZE; fix that assumption. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index ab9746aff766..78f888d028c5 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -248,7 +248,7 @@ static void page_cache_free_page(struct address_space *mapping, freepage(page); if (PageTransHuge(page) && !PageHuge(page)) { - page_ref_sub(page, HPAGE_PMD_NR); + page_ref_sub(page, thp_nr_pages(page)); VM_BUG_ON_PAGE(page_count(page) <= 0, page); } else { put_page(page); From patchwork Wed Jun 10 20:13:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598823 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9B0A190 for ; Wed, 10 Jun 2020 20:39:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6779420814 for ; Wed, 10 Jun 2020 20:39:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="LAoTgmZq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6779420814 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5725F8D001F; Wed, 10 Jun 2020 16:39:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4F77C8D000E; Wed, 10 Jun 2020 16:39:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3990F8D001F; Wed, 10 Jun 2020 16:39:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id 21E2A8D000E for ; Wed, 10 Jun 2020 16:39:15 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D4E1E181ABEA4 for ; Wed, 10 Jun 2020 20:39:14 +0000 (UTC) X-FDA: 76914467028.05.stew87_4b0515426dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id ACBFB18036FEE for ; Wed, 10 Jun 2020 20:39:14 +0000 (UTC) X-Spam-Summary: 2,0,0,c3c7cfe225656087,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1540:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2693:2892:3138:3139:3140:3141:3142:3352:3865:3866:3867:3870:4250:4321:5007:6119:6261:6653:7576:8603:10004:11026:11473:11658:11914:12296:12297:12438:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21451:21627:21990:30003:30054:30070:30075,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: stew87_4b0515426dce X-Filterd-Recvd-Size: 2555 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=W4JF/8+4T74bu7dBYV5gPvgjn/5tVoyv2zfUGPd36qk=; b=LAoTgmZqcjMuG+HCWwA5BxKMbm Nyiwbqf3Xa8NAh4iH5iThagnkB86tvgqE8VqwdsvILSa9ApozrBRx1UoRfXdTZY/Cauy6TKA4zr34 0CObYbAOF5vT9G/R3029RFDiAZxL7qUl2BM9V7U2lxXmSofogsIyKWzY47bfxN6z4gYrOIMB6jnJQ XvDcxJqlghi4i4a7SL7APlV+VMSh5Xnl9f/Wc9y0kJ+PwBpwV6MZx2UdiRSoCuxEePDsAKtEzj6Cn 2cIpvBS3CHQbmUe22So5PS0HyRXQutAimltARhNfCjSbgayoieOylYetLtrB+ZvtZS2bF0Gu6vs+2 bqKSLVmw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003Wz-P8; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 37/51] mm: Remove page fault assumption of compound page size Date: Wed, 10 Jun 2020 13:13:31 -0700 Message-Id: <20200610201345.13273-38-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: ACBFB18036FEE X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" A compound page in the page cache will not necessarily be of PMD size, so check explicitly. Signed-off-by: Matthew Wilcox (Oracle) --- mm/memory.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index dc7f3543b1fd..3e6ef0ebfdd0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3552,13 +3552,14 @@ static vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) unsigned long haddr = vmf->address & HPAGE_PMD_MASK; pmd_t entry; int i; - vm_fault_t ret; + vm_fault_t ret = VM_FAULT_FALLBACK; if (!transhuge_vma_suitable(vma, haddr)) - return VM_FAULT_FALLBACK; + return ret; - ret = VM_FAULT_FALLBACK; page = compound_head(page); + if (page_order(page) != HPAGE_PMD_ORDER) + return ret; /* * Archs like ppc64 need additonal space to store information From patchwork Wed Jun 10 20:13:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598849 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AE487618 for ; Wed, 10 Jun 2020 20:39:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7AD95206A4 for ; Wed, 10 Jun 2020 20:39:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="c2zxVuJH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7AD95206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 261F18D0028; Wed, 10 Jun 2020 16:39:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1A6978D002D; Wed, 10 Jun 2020 16:39:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3B0B8D002C; Wed, 10 Jun 2020 16:39:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0183.hostedemail.com [216.40.44.183]) by kanga.kvack.org (Postfix) with ESMTP id AEF8F8D0028 for ; Wed, 10 Jun 2020 16:39:28 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 6D03A181ABEA4 for ; Wed, 10 Jun 2020 20:39:28 +0000 (UTC) X-FDA: 76914467616.10.screw15_4f14e2726dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 40C1A1B3F8 for ; Wed, 10 Jun 2020 20:39:28 +0000 (UTC) X-Spam-Summary: 2,0,0,4fdaa10161037ed9,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:2731:3138:3139:3140:3141:3142:3352:3866:3870:4321:5007:6117:6261:6653:7576:7875:8957:9010:10004:11026:11473:11658:11914:12043:12296:12297:12555:12895:12986:13069:13184:13229:13311:13357:13894:14181:14384:14394:14721:21080:21451:21627:21990:30054:30090,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: screw15_4f14e2726dce X-Filterd-Recvd-Size: 2939 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf29.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=034RyJserNfH7yl0tdVgBoooxMtosP1kfL7i1H9r4Ho=; b=c2zxVuJHjKpbwjap05D0eIWUuf So6yrmIFHNfcd0/YkqoqRdNzrSm5Yzhrv3np0BifU/TqiIIU4SlbWEMLjymXH5QOO34gZES0HKd61 E/oxVQOxW3v/zbwSh7/B7LoYArFbIbfQQ/L+NI1wKEMnxMpl2E4ZIlupIXkuM+oINJOwpHE3T3bgr 5ZuzTR6yEKtesYzttadRPP1DQIX407Hf5FK0hH8mPtF33TIFf5dJl+fHzybXoT8APTDNCSO7+8YmR uLAK8L+MJLGtnBP45zhZXC0IqOid40u31a2mwsAE8VNvUvdyjFQAmKSyedxNSRO4P4/KZFlLXcZG0 lL3pO8zA==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003X5-Qw; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Kirill A. Shutemov" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Matthew Wilcox Subject: [PATCH v6 38/51] mm: Fix total_mapcount assumption of page size Date: Wed, 10 Jun 2020 13:13:32 -0700 Message-Id: <20200610201345.13273-39-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 40C1A1B3F8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Kirill A. Shutemov" File THPs may now be of arbitrary order. Signed-off-by: Kirill A. Shutemov Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 80fb38e93837..744863aa0374 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2490,7 +2490,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, int total_mapcount(struct page *page) { - int i, compound, ret; + int i, compound, nr, ret; VM_BUG_ON_PAGE(PageTail(page), page); @@ -2498,16 +2498,17 @@ int total_mapcount(struct page *page) return atomic_read(&page->_mapcount) + 1; compound = compound_mapcount(page); + nr = compound_nr(page); if (PageHuge(page)) return compound; ret = compound; - for (i = 0; i < HPAGE_PMD_NR; i++) + for (i = 0; i < nr; i++) ret += atomic_read(&page[i]._mapcount) + 1; /* File pages has compound_mapcount included in _mapcount */ if (!PageAnon(page)) - return ret - compound * HPAGE_PMD_NR; + return ret - compound * nr; if (PageDoubleMap(page)) - ret -= HPAGE_PMD_NR; + ret -= nr; return ret; } From patchwork Wed Jun 10 20:13:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598845 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5D6CD90 for ; Wed, 10 Jun 2020 20:39:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2072F207ED for ; Wed, 10 Jun 2020 20:39:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="qA6a873s" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2072F207ED Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 08DA88D002A; Wed, 10 Jun 2020 16:39:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F37358D0028; Wed, 10 Jun 2020 16:39:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C73FF8D002B; Wed, 10 Jun 2020 16:39:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0062.hostedemail.com [216.40.44.62]) by kanga.kvack.org (Postfix) with ESMTP id A66878D0028 for ; Wed, 10 Jun 2020 16:39:27 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 62A78181ABEAA for ; Wed, 10 Jun 2020 20:39:27 +0000 (UTC) X-FDA: 76914467574.26.jelly63_1b16d3c26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 350A218027662 for ; Wed, 10 Jun 2020 20:39:27 +0000 (UTC) X-Spam-Summary: 2,0,0,8ea60fcaee1ba753,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1605:1711:1730:1747:1777:1792:2393:2559:2562:2898:3138:3139:3140:3141:3142:3865:3866:3870:3871:3874:4250:4321:4419:4605:5007:6119:6120:6261:6653:7576:7875:7901:7903:8957:9010:10004:11026:11473:11658:11914:12043:12296:12297:12438:12555:12895:12986:13894:14181:14394:14721:21080:21451:21627:21990:30054:30070,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:0,LUA_SUMMARY:none X-HE-Tag: jelly63_1b16d3c26dce X-Filterd-Recvd-Size: 5947 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=K+NqwcuwBBsF5gvk4uOfKfFisJ5O9Bzckjl4rjWXvjY=; b=qA6a873s/Zff6UrlIw+ycMtkmr r7DXIh0LqRMxP8WmkQDOXoK4uBozwknJV1jplSMSs98TIdUIx9XWMUHtY5l3pS6At6H0Xo3WrxqxB Hd4PKKJiOVr/UNzXW9P1ADiQ2BXcPTKbHqHw4Vu3jZHP/lMxn1vwvo14aDBfSs/nxM5xlnqoahpLH bOivbzFTqd30i9dX6D36QV9MDuQJbVOHE7P8sXzMHd/HO5kvplNVn8doNMTZ7o47aCSZFxVTtjAEa kwGXfT5L3TCXUpfoZOZ8sSz+qHMzHK1b84OTzHcsHm1cPxLrUaCSX1eV8+lK+rgZy7PRnoEdC+vgv paZTSwUA==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003XE-SA; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 39/51] mm: Remove assumptions of THP size Date: Wed, 10 Jun 2020 13:13:33 -0700 Message-Id: <20200610201345.13273-40-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 350A218027662 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Remove direct uses of HPAGE_PMD_NR in paths that aren't necessarily PMD sized. Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 15 ++++++++------- mm/rmap.c | 10 +++++----- 2 files changed, 13 insertions(+), 12 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 744863aa0374..c25d8e2310e8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2340,7 +2340,7 @@ static void remap_page(struct page *page) if (PageTransHuge(page)) { remove_migration_ptes(page, page, true); } else { - for (i = 0; i < HPAGE_PMD_NR; i++) + for (i = 0; i < thp_nr_pages(page); i++) remove_migration_ptes(page + i, page + i, true); } } @@ -2415,6 +2415,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, struct lruvec *lruvec; struct address_space *swap_cache = NULL; unsigned long offset = 0; + unsigned int nr = thp_nr_pages(head); int i; lruvec = mem_cgroup_page_lruvec(head, pgdat); @@ -2430,7 +2431,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, xa_lock(&swap_cache->i_pages); } - for (i = HPAGE_PMD_NR - 1; i >= 1; i--) { + for (i = nr - 1; i >= 1; i--) { __split_huge_page_tail(head, i, lruvec, list); /* Some pages can be beyond i_size: drop them from page cache */ if (head[i].index >= end) { @@ -2471,7 +2472,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, remap_page(head); - for (i = 0; i < HPAGE_PMD_NR; i++) { + for (i = 0; i < nr; i++) { struct page *subpage = head + i; if (subpage == page) continue; @@ -2553,14 +2554,14 @@ int page_trans_huge_mapcount(struct page *page, int *total_mapcount) page = compound_head(page); _total_mapcount = ret = 0; - for (i = 0; i < HPAGE_PMD_NR; i++) { + for (i = 0; i < thp_nr_pages(page); i++) { mapcount = atomic_read(&page[i]._mapcount) + 1; ret = max(ret, mapcount); _total_mapcount += mapcount; } if (PageDoubleMap(page)) { ret -= 1; - _total_mapcount -= HPAGE_PMD_NR; + _total_mapcount -= thp_nr_pages(page); } mapcount = compound_mapcount(page); ret += mapcount; @@ -2577,9 +2578,9 @@ bool can_split_huge_page(struct page *page, int *pextra_pins) /* Additional pins from page cache */ if (PageAnon(page)) - extra_pins = PageSwapCache(page) ? HPAGE_PMD_NR : 0; + extra_pins = PageSwapCache(page) ? thp_nr_pages(page) : 0; else - extra_pins = HPAGE_PMD_NR; + extra_pins = thp_nr_pages(page); if (pextra_pins) *pextra_pins = extra_pins; return total_mapcount(page) == page_count(page) - extra_pins - 1; diff --git a/mm/rmap.c b/mm/rmap.c index c56fab5826c1..c0295282928b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1205,7 +1205,7 @@ void page_add_file_rmap(struct page *page, bool compound) VM_BUG_ON_PAGE(compound && !PageTransHuge(page), page); lock_page_memcg(page); if (compound && PageTransHuge(page)) { - for (i = 0, nr = 0; i < HPAGE_PMD_NR; i++) { + for (i = 0, nr = 0; i < thp_nr_pages(page); i++) { if (atomic_inc_and_test(&page[i]._mapcount)) nr++; } @@ -1246,7 +1246,7 @@ static void page_remove_file_rmap(struct page *page, bool compound) /* page still mapped by someone else? */ if (compound && PageTransHuge(page)) { - for (i = 0, nr = 0; i < HPAGE_PMD_NR; i++) { + for (i = 0, nr = 0; i < thp_nr_pages(page); i++) { if (atomic_add_negative(-1, &page[i]._mapcount)) nr++; } @@ -1293,7 +1293,7 @@ static void page_remove_anon_compound_rmap(struct page *page) * Subpages can be mapped with PTEs too. Check how many of * them are still mapped. */ - for (i = 0, nr = 0; i < HPAGE_PMD_NR; i++) { + for (i = 0, nr = 0; i < thp_nr_pages(page); i++) { if (atomic_add_negative(-1, &page[i]._mapcount)) nr++; } @@ -1303,10 +1303,10 @@ static void page_remove_anon_compound_rmap(struct page *page) * page of the compound page is unmapped, but at least one * small page is still mapped. */ - if (nr && nr < HPAGE_PMD_NR) + if (nr && nr < thp_nr_pages(page)) deferred_split_huge_page(page); } else { - nr = HPAGE_PMD_NR; + nr = thp_nr_pages(page); } if (unlikely(PageMlocked(page))) From patchwork Wed Jun 10 20:13:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598857 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DE7C590 for ; Wed, 10 Jun 2020 20:39:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AAC40206A4 for ; Wed, 10 Jun 2020 20:39:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="S8TmZlwJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AAC40206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EBFB68D002F; Wed, 10 Jun 2020 16:39:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DD5058D002D; Wed, 10 Jun 2020 16:39:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F1EA8D002F; Wed, 10 Jun 2020 16:39:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0001.hostedemail.com [216.40.44.1]) by kanga.kvack.org (Postfix) with ESMTP id 735588D002D for ; Wed, 10 Jun 2020 16:39:31 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 363FB8017CA1 for ; Wed, 10 Jun 2020 20:39:31 +0000 (UTC) X-FDA: 76914467742.25.paste17_2e0990926dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 035141804E3E1 for ; Wed, 10 Jun 2020 20:39:30 +0000 (UTC) X-Spam-Summary: 2,0,0,a0be1b81fdb3cda8,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1540:1711:1730:1747:1777:1792:2393:2559:2562:2895:3138:3139:3140:3141:3142:3352:3865:3867:4321:5007:6119:6261:6653:7576:8957:10004:11026:11658:11914:12043:12296:12297:12438:12555:12895:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21451:21627:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: paste17_2e0990926dce X-Filterd-Recvd-Size: 2443 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=g1BN8ZEjBZSGzNik3H7y+eWzpa3p549rVRPP5tfDWW4=; b=S8TmZlwJGr4i53ZlyGiAARSx3B o/PTdqvC3Swsl/OTVTcOgVe3eSV6K/0lN7UW/J5JPqo+tAU8viOyJPIKIOu1hsqEeVXLnXrnUi2Vs YLJKG7xytHER10sJZqEc+TF9ikz4FxfiSvF3JS43qvLHLPlZ3caBAP88T39Gt9/lotHFCNCRWbwRZ yP+Q39vBRcNTIIwZfQUSvkyouf7jnoCZ/u2TFOE5fymJM/9gqbluTgEgio3WiJrQI6OqXs/LuecAX 0oxXAbq9/a4ROH7pvpiQXLA3D209RsVrkiXL78R/X4E7CwcOcy1t67dxdH3a88p/G+B9aWnTMXHsf cCP91qEA==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003XI-TB; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 40/51] mm: Avoid splitting THPs Date: Wed, 10 Jun 2020 13:13:34 -0700 Message-Id: <20200610201345.13273-41-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 035141804E3E1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" If the filesystem supports THPs, then do not split them before removing them from the page cache; remove them as a unit. --- mm/vmscan.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 17934e03b3aa..0db62c1001f7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1277,9 +1277,9 @@ static unsigned int shrink_page_list(struct list_head *page_list, /* Adding to swap updated mapping */ mapping = page_mapping(page); } - } else if (unlikely(PageTransHuge(page))) { - /* Split file THP */ - if (split_huge_page_to_list(page, page_list)) + } else if (PageTransHuge(page)) { + if ((!mapping || !mapping_thp_support(mapping)) && + split_huge_page_to_list(page, page_list)) goto keep_locked; } From patchwork Wed Jun 10 20:13:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598855 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EE46C90 for ; Wed, 10 Jun 2020 20:39:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BA4E1206A4 for ; Wed, 10 Jun 2020 20:39:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Q5FK+pK0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA4E1206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8F5358D0031; Wed, 10 Jun 2020 16:39:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 856558D0030; Wed, 10 Jun 2020 16:39:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 631FF8D002F; Wed, 10 Jun 2020 16:39:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0218.hostedemail.com [216.40.44.218]) by kanga.kvack.org (Postfix) with ESMTP id 414A78D002D for ; Wed, 10 Jun 2020 16:39:31 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0D9711801A818 for ; Wed, 10 Jun 2020 20:39:31 +0000 (UTC) X-FDA: 76914467742.26.hook85_0b0cbd326dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id D87561807ECF4 for ; Wed, 10 Jun 2020 20:39:30 +0000 (UTC) X-Spam-Summary: 2,0,0,5ff585434e741206,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:2693:3138:3139:3140:3141:3142:3353:3865:3867:3868:3871:3872:3874:4250:4321:4419:5007:6261:6653:7576:7903:8957:10004:11026:11473:11658:11914:12296:12297:12438:12555:12895:12986:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21324:21451:21627:21795:30051:30054:30090,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: hook85_0b0cbd326dce X-Filterd-Recvd-Size: 3165 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf47.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=mmPoGnRfA4TUBSdV1ap2zn7Ew2a8qCnx8Rp9BKBKl4s=; b=Q5FK+pK0rlhrqdQfcRpYaQNhyr 9ERE6UdAQr92hPMbjQZaSR5ylWRW5PjUBvl0S98MdVV1cmRWTQBfI6DjgaHA5QgTMk9o+4YZefDFQ ujIFF9I4t50ykVeBiDedoUZ98B96D3Op4PnbePR9YyBhA4+9IK9LlhSQPjg9/RQRUIEfRGuxaQJAX g2GrClrm5J9SRCH2mm1DlmaO59hh4JEpYl4kFUkvT2KKIJRMU53JdILUluaWtzQVQDk4CoWxhkup/ oXlPFQLp6+orAWifK/R6CoiLOnDCCYy0IPybyQA2HShydsk9CInMgFQpxox701IJEFOl0z1LTamw8 9FJaHaaw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003XP-UF; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 41/51] mm: Fix truncation for pages of arbitrary size Date: Wed, 10 Jun 2020 13:13:35 -0700 Message-Id: <20200610201345.13273-42-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D87561807ECF4 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Remove the assumption that a compound page is HPAGE_PMD_SIZE, and the assumption that any page is PAGE_SIZE. Signed-off-by: Matthew Wilcox (Oracle) --- mm/truncate.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/truncate.c b/mm/truncate.c index 152974888124..a9fde773179b 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -168,7 +168,7 @@ void do_invalidatepage(struct page *page, unsigned int offset, * becomes orphaned. It will be left on the LRU and may even be mapped into * user pagetables if we're racing with filemap_fault(). * - * We need to bale out if page->mapping is no longer equal to the original + * We need to bail out if page->mapping is no longer equal to the original * mapping. This happens a) when the VM reclaimed the page while we waited on * its lock, b) when a concurrent invalidate_mapping_pages got there first and * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space. @@ -177,12 +177,12 @@ static void truncate_cleanup_page(struct address_space *mapping, struct page *page) { if (page_mapped(page)) { - pgoff_t nr = PageTransHuge(page) ? HPAGE_PMD_NR : 1; + unsigned int nr = thp_nr_pages(page); unmap_mapping_pages(mapping, page->index, nr, false); } if (page_has_private(page)) - do_invalidatepage(page, 0, PAGE_SIZE); + do_invalidatepage(page, 0, thp_size(page)); /* * Some filesystems seem to re-dirty the page even after From patchwork Wed Jun 10 20:13:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598861 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8541690 for ; Wed, 10 Jun 2020 20:39:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 48282206A4 for ; Wed, 10 Jun 2020 20:39:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Ui7p86en" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 48282206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 23CC08D0032; Wed, 10 Jun 2020 16:39:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 14E768D002D; Wed, 10 Jun 2020 16:39:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E20FA8D0033; Wed, 10 Jun 2020 16:39:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0170.hostedemail.com [216.40.44.170]) by kanga.kvack.org (Postfix) with ESMTP id C5F228D0032 for ; Wed, 10 Jun 2020 16:39:32 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 81DBB148161 for ; Wed, 10 Jun 2020 20:39:32 +0000 (UTC) X-FDA: 76914467784.23.pie98_561444c26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 5A70C17EBD for ; Wed, 10 Jun 2020 20:39:32 +0000 (UTC) X-Spam-Summary: 2,0,0,af9bb29e63afafb8,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:2:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1606:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2731:2898:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:4118:4250:4321:4384:4385:4395:5007:6119:6261:6653:7576:7875:8603:8660:8957:9592:10004:11026:11473:11658:11914:12043:12291:12295:12296:12297:12438:12555:12683:12895:12986:13141:13148:13230:13894:14394:21080:21324:21451:21627:21939:21990:30054:30070,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: pie98_561444c26dce X-Filterd-Recvd-Size: 7519 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=YTIkvzPZ9eKKWGpmM1bIB6WPTuavuZdTbpKhhYx4bUI=; b=Ui7p86en44wnRWdlztqtIrnIXp QnVpAEmonyzEDvD13FWu9/MFQcrcpHPf/FCNY4AYnqRz3Da27rpouQGMbd1Y/caEpdSG3LaRRURtb cfKcfjB8DzZvwBOijT1fjKCvF8VYUEZS5TpaD0WMEiZmmAcNuZP4PXZ3Lr3cMPbPY50TvWMiSGPMD wYVoMINjh55MPviyrXRvtVbm6KeGHtv7rTRalzDdWS9sNcLN6C8Mgkqu4DAqCgpMaR2L8XDYD7KO7 vPARQVRCy3zjcqwzlCNarjGk4gh/xdBD+qSRLa7jmdDGwxH3iA21vNAux1j0H7xsWzF2CnQniDz9+ VVBqK5ag==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003XW-Vv; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 42/51] mm: Handle truncates that split THPs Date: Wed, 10 Jun 2020 13:13:36 -0700 Message-Id: <20200610201345.13273-43-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5A70C17EBD X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Move shmem_punch_compound() to truncate.c and rename it to punch_thp(). Change its arguments to loff_t to make calling do_invalidatepage() easier. Call it when we find a THP in the cache. Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 2 ++ mm/shmem.c | 30 ++------------------------- mm/truncate.c | 57 +++++++++++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 59 insertions(+), 30 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index ac3c79408045..cd7038a36354 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -613,4 +613,6 @@ static inline bool is_migrate_highatomic_page(struct page *page) void setup_zone_pageset(struct zone *zone); extern struct page *alloc_new_node_page(struct page *page, unsigned long node); + +bool punch_thp(struct page *page, loff_t start, loff_t end); #endif /* __MM_INTERNAL_H */ diff --git a/mm/shmem.c b/mm/shmem.c index 55405d811cfd..495b8684d94a 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -804,32 +804,6 @@ void shmem_unlock_mapping(struct address_space *mapping) } } -/* - * Check whether a hole-punch or truncation needs to split a huge page, - * returning true if no split was required, or the split has been successful. - * - * Eviction (or truncation to 0 size) should never need to split a huge page; - * but in rare cases might do so, if shmem_undo_range() failed to trylock on - * head, and then succeeded to trylock on tail. - * - * A split can only succeed when there are no additional references on the - * huge page: so the split below relies upon find_get_entries() having stopped - * when it found a subpage of the huge page, without getting further references. - */ -static bool shmem_punch_compound(struct page *page, pgoff_t start, pgoff_t end) -{ - if (!PageTransCompound(page)) - return true; - - /* Just proceed to delete a huge page wholly within the range punched */ - if (PageHead(page) && - page->index >= start && page->index + HPAGE_PMD_NR <= end) - return true; - - /* Try to split huge page, so we can truly punch the hole or truncate */ - return split_huge_page(page) >= 0; -} - /* * Remove range of pages and swap entries from page cache, and free them. * If !unfalloc, truncate or punch hole; if unfalloc, undo failed fallocate. @@ -883,7 +857,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, if ((!unfalloc || !PageUptodate(page)) && page_mapping(page) == mapping) { VM_BUG_ON_PAGE(PageWriteback(page), page); - if (shmem_punch_compound(page, start, end)) + if (punch_thp(page, lstart, lend)) truncate_inode_page(mapping, page); } unlock_page(page); @@ -973,7 +947,7 @@ static void shmem_undo_range(struct inode *inode, loff_t lstart, loff_t lend, break; } VM_BUG_ON_PAGE(PageWriteback(page), page); - if (shmem_punch_compound(page, start, end)) + if (punch_thp(page, lstart, lend)) truncate_inode_page(mapping, page); else if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) { /* Wipe the page and don't get stuck */ diff --git a/mm/truncate.c b/mm/truncate.c index a9fde773179b..0ef2001c2f65 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -229,6 +229,55 @@ int truncate_inode_page(struct address_space *mapping, struct page *page) return 0; } +/* + * Check whether a hole-punch or truncation needs to split a huge page, + * returning true if no split was required, or the split has been + * successful. + * + * Eviction (or truncation to 0 size) should never need to split a huge + * page; but in rare cases might do so, if shmem_undo_range() failed to + * trylock on head, and then succeeded to trylock on tail. + * + * A split can only succeed when there are no additional references on + * the huge page: so the split below relies upon find_get_entries() + * having stopped when it found a subpage of the huge page, without + * getting further references. + */ +bool punch_thp(struct page *page, loff_t start, loff_t end) +{ + struct page *head = thp_head(page); + loff_t pos = page_offset(head); + unsigned int offset, length; + + if (!PageTransCompound(page)) + return true; + + if (pos < start) + offset = start - pos; + else + offset = 0; + length = thp_size(head); + if (pos + length < end) + length = length - offset; + else + length = end - pos - offset; + + /* Just proceed to delete a huge page wholly within the range punched */ + if (length == thp_size(head)) + return true; + + /* + * We're going to split the page into order-0 pages. Tell the + * filesystem which range of the page is going to be punched out + * so it can discard unnecessary private data. + */ + if (page_has_private(head)) + do_invalidatepage(head, offset, length); + + /* Try to split huge page, so we can truly punch the hole or truncate */ + return split_huge_page(page) >= 0; +} + /* * Used to get rid of pages on hardware memory corruption. */ @@ -359,7 +408,10 @@ void truncate_inode_pages_range(struct address_space *mapping, unlock_page(page); continue; } - pagevec_add(&locked_pvec, page); + if (punch_thp(page, lstart, lend)) + pagevec_add(&locked_pvec, page); + else + unlock_page(page); } for (i = 0; i < pagevec_count(&locked_pvec); i++) truncate_cleanup_page(mapping, locked_pvec.pages[i]); @@ -453,7 +505,8 @@ void truncate_inode_pages_range(struct address_space *mapping, lock_page(page); WARN_ON(page_to_index(page) != index); wait_on_page_writeback(page); - truncate_inode_page(mapping, page); + if (punch_thp(page, lstart, lend)) + truncate_inode_page(mapping, page); unlock_page(page); } truncate_exceptional_pvec_entries(mapping, &pvec, indices, end); From patchwork Wed Jun 10 20:13:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598811 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2217F90 for ; Wed, 10 Jun 2020 20:39:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E2A4E206A4 for ; Wed, 10 Jun 2020 20:39:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="hwWRASta" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E2A4E206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 28AC08D0019; Wed, 10 Jun 2020 16:39:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 213C48D000E; Wed, 10 Jun 2020 16:39:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0F208D0019; Wed, 10 Jun 2020 16:39:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0145.hostedemail.com [216.40.44.145]) by kanga.kvack.org (Postfix) with ESMTP id D90F18D000E for ; Wed, 10 Jun 2020 16:39:03 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 98B5C180AEF50 for ; Wed, 10 Jun 2020 20:39:03 +0000 (UTC) X-FDA: 76914466566.25.mist28_380d4e126dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 665DD1804E3E1 for ; Wed, 10 Jun 2020 20:39:03 +0000 (UTC) X-Spam-Summary: 2,0,0,d9e123e8876dd4a2,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:3870:3871:3872:3874:4321:5007:6261:6653:7576:7903:8957:10004:11026:11232:11473:11658:11914:12043:12296:12297:12555:12895:12986:13069:13311:13357:13894:14096:14181:14384:14394:14721:21080:21324:21451:21627:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:0,LUA_SUMMARY:none X-HE-Tag: mist28_380d4e126dce X-Filterd-Recvd-Size: 3141 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=WQYUHtaa1PlP1XkIa4APm8ZxDaohf3g/uelkXzjDEzk=; b=hwWRASta1tlG6WCdBl2KkJNJzr nvbYBTfI6Mu/K4m6gQEQMPCt4LO80m/FyuVq9tIMkJ18nlag0tghEzd0QiTIVzu0lZs0rset5WvKJ N5pJwzizSEwx+sOoPLBmjq8PrfTpbD7VTsJyaqD0x12p/HXI0yJPHzDqZo/Evn/z1uxGFBUvcHDVa UTyq1eTa+oiGoOwpFCZNYT+kWRClHGiofccsEnWMUQkHGAuKD8oKJPG52/BwQ3KvVEBHb4LlvZM9L HP0YPkLSaqemnXdZLxKTisKN+6Sj48890bydbdM0IWKzEuXCLJvnL1guhhfpRkJjy2LIRWjiFXayR FPX2gcfQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76b-0003Xd-0l; Wed, 10 Jun 2020 20:13:49 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 43/51] mm: Support storing shadow entries for THPs Date: Wed, 10 Jun 2020 13:13:37 -0700 Message-Id: <20200610201345.13273-44-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 665DD1804E3E1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" If the page is being replaced with a NULL, we can do a single store that erases the entire range of indices. Otherwise we have to use a loop to store one shadow entry in each index. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 78f888d028c5..17db007f0277 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -120,22 +120,27 @@ static void page_cache_delete(struct address_space *mapping, struct page *page, void *shadow) { XA_STATE(xas, &mapping->i_pages, page->index); - unsigned int nr = 1; + unsigned int i, nr = 1, entries = 1; mapping_set_update(&xas, mapping); /* hugetlb pages are represented by a single entry in the xarray */ if (!PageHuge(page)) { - xas_set_order(&xas, page->index, compound_order(page)); - nr = compound_nr(page); + entries = nr = thp_nr_pages(page); + if (!shadow) { + xas_set_order(&xas, page->index, thp_order(page)); + entries = 1; + } } VM_BUG_ON_PAGE(!PageLocked(page), page); VM_BUG_ON_PAGE(PageTail(page), page); - VM_BUG_ON_PAGE(nr != 1 && shadow, page); - xas_store(&xas, shadow); - xas_init_marks(&xas); + for (i = 0; i < entries; i++) { + xas_store(&xas, shadow); + xas_init_marks(&xas); + xas_next(&xas); + } page->mapping = NULL; /* Leave page->index set: truncation lookup relies upon it */ From patchwork Wed Jun 10 20:13:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598805 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0CA0E90 for ; Wed, 10 Jun 2020 20:39:01 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C2A6D20825 for ; Wed, 10 Jun 2020 20:39:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Pbb6eP2r" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C2A6D20825 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id EE4BB8D0016; Wed, 10 Jun 2020 16:38:59 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E6C3B8D000E; Wed, 10 Jun 2020 16:38:59 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5B438D0016; Wed, 10 Jun 2020 16:38:59 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0132.hostedemail.com [216.40.44.132]) by kanga.kvack.org (Postfix) with ESMTP id BB16D8D000E for ; Wed, 10 Jun 2020 16:38:59 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8653E801B442 for ; Wed, 10 Jun 2020 20:38:59 +0000 (UTC) X-FDA: 76914466398.02.value98_25010cd26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 5A3F9160D9 for ; Wed, 10 Jun 2020 20:38:59 +0000 (UTC) X-Spam-Summary: 2,0,0,a7b988a064e58de4,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1539:1568:1711:1714:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3867:3870:3871:5007:6261:6653:7576:10004:11026:11232:11473:11658:11914:12043:12296:12297:12555:12895:13069:13311:13357:13894:14181:14384:14394:14721:21080:21451:21627:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:22,LUA_SUMMARY:none X-HE-Tag: value98_25010cd26dce X-Filterd-Recvd-Size: 2323 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:38:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=N4mF7LsOQy9p4C9kHzIXaco2T4XYfdDlUPlC4E84MzA=; b=Pbb6eP2rMdef2upRDhntGlFJFl IIHSHixhHLaxiEd6ofKCTfBsbQ/+y1OzZB2K1PbAyb3Ia66m+/QoJNRHj+zYjM2ORzsxtO0j1cNtU 6vmM/l0XviygY97jTLFGdEZ59HXDgvDLExHLt9xYIyRdhdpNG8eADURaLxV7kAIG34vxwcw7hzSU0 Jsjk7xZgQ4uyEjjgS9WSyA+u1aKrSmHmkHeAuaZ3MlXIxZHuI40FWwCeKfV3K03yBZAgy9g6iLZ45 qKCqdO2D/5WbMi5iRxQS+8Ah1Ws1HFtamZwPs2dJUY63FAn2yYlx31B5vNQokJ1eyUh9xRr1VmikZ bCJEQmVw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76b-0003Xk-2c; Wed, 10 Jun 2020 20:13:49 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 44/51] mm: Support retrieving tail pages from the page cache Date: Wed, 10 Jun 2020 13:13:38 -0700 Message-Id: <20200610201345.13273-45-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5A3F9160D9 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" page->index is not meaningful for tail pages; we have to use page_to_index() in this assertion. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index 17db007f0277..869b970fe1ab 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1652,7 +1652,7 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t index, put_page(page); goto repeat; } - VM_BUG_ON_PAGE(page->index != index, page); + VM_BUG_ON_PAGE(page_to_index(page) != index, page); } if (fgp_flags & FGP_ACCESSED) From patchwork Wed Jun 10 20:13:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598819 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BEAEF618 for ; Wed, 10 Jun 2020 20:39:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8198B207ED for ; Wed, 10 Jun 2020 20:39:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="oDy2sEk0" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8198B207ED Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F33988D001D; Wed, 10 Jun 2020 16:39:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EBAE38D000E; Wed, 10 Jun 2020 16:39:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE8028D001D; Wed, 10 Jun 2020 16:39:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0212.hostedemail.com [216.40.44.212]) by kanga.kvack.org (Postfix) with ESMTP id A7EE08D000E for ; Wed, 10 Jun 2020 16:39:12 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 6AEA5181ABEA4 for ; Wed, 10 Jun 2020 20:39:12 +0000 (UTC) X-FDA: 76914466944.16.steam87_3a1197226dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 3F3F1100596E0 for ; Wed, 10 Jun 2020 20:39:12 +0000 (UTC) X-Spam-Summary: 2,0,0,8aca5c348afd31e5,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1539:1567:1711:1714:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3865:3871:5007:6261:6653:7576:10004:11026:11473:11658:11914:12043:12114:12297:12438:12555:12895:13069:13161:13229:13311:13357:13894:14181:14384:14394:14721:21080:21324:21451:21627:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: steam87_3a1197226dce X-Filterd-Recvd-Size: 2292 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf16.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=hjtxBqocMNhYkpp8Xg/ALyMhtyMIq4PqDFO3IKqKJpw=; b=oDy2sEk0C4TA1UDvwaLLIZI1pR JOfaoKJP5U4cmw6dzVSzyT3+5dHoqA5/I+266eum7s4YMxumC41j4BEMSKdVFxg0rqbQ8EbzxBcW0 tmcsUU+gXd4itwdzOvXJ5v/MZm8kd3gIKt+aNO9QygBxeQtig1m/7iTG4ECvpESE77noHDK+IW6z/ RWTDg2rHeESxtEQ+zT5ERB5MxcdUm10gFtU4uXzT6dJeOzmjFm7ALjQgKjbnjA3BIGOttq2nL0j8H Ol3SEWEzBDX1QHTfee6yb1WHMFl6UnuTzt1vzj5CoH3AmzGf6wiJnf5AqBFcKttjctcP7bhSRSrfN QhauW/9Q==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76b-0003Xr-3g; Wed, 10 Jun 2020 20:13:49 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 45/51] mm: Support tail pages in wait_for_stable_page Date: Wed, 10 Jun 2020 13:13:39 -0700 Message-Id: <20200610201345.13273-46-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3F3F1100596E0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" page->mapping is undefined for tail pages, so operate exclusively on the head page. Signed-off-by: Matthew Wilcox (Oracle) --- mm/page-writeback.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/page-writeback.c b/mm/page-writeback.c index 28b3e7a67565..1b358aac065f 100644 --- a/mm/page-writeback.c +++ b/mm/page-writeback.c @@ -2851,6 +2851,7 @@ EXPORT_SYMBOL_GPL(wait_on_page_writeback); */ void wait_for_stable_page(struct page *page) { + page = thp_head(page); if (bdi_cap_stable_pages_required(inode_to_bdi(page->mapping->host))) wait_on_page_writeback(page); } From patchwork Wed Jun 10 20:13:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598807 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DBD2190 for ; Wed, 10 Jun 2020 20:39:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A82A42081A for ; Wed, 10 Jun 2020 20:39:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="B6YqyDI1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A82A42081A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4850F8D0017; Wed, 10 Jun 2020 16:39:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3E6718D000E; Wed, 10 Jun 2020 16:39:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19F148D0017; Wed, 10 Jun 2020 16:39:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0134.hostedemail.com [216.40.44.134]) by kanga.kvack.org (Postfix) with ESMTP id EFFD88D000E for ; Wed, 10 Jun 2020 16:39:00 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id AE2401E097 for ; Wed, 10 Jun 2020 20:39:00 +0000 (UTC) X-FDA: 76914466440.20.girls45_4b03f2b26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 8796F1801BF71 for ; Wed, 10 Jun 2020 20:39:00 +0000 (UTC) X-Spam-Summary: 2,0,0,e196f532e7d7792e,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3866:3867:4321:4605:5007:6261:6653:7576:10004:11026:11658:11914:12043:12296:12297:12438:12555:12895:13069:13311:13357:13894:14181:14384:14394:14721:21080:21451:21627:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:15,LUA_SUMMARY:none X-HE-Tag: girls45_4b03f2b26dce X-Filterd-Recvd-Size: 2904 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=nogVZwCGMHTUnccVf6su9Crc41Rj8q29sZmEfeg+55U=; b=B6YqyDI1W81riSbux+Vqd6NQmq cAoL/rxz0dLAReAf6MspJHD1x1Gb9CYEfTcjhnFjQuAWU+a3aOM3Gq3E1DLA7K2peW0+M7zBhIOyX K76vx756jdjTDWtlW56VZupLDFsFQl2sC34HjPV/uiS1BmthqJmt9VWcNr+Y6FMNHCDGgGX/pLFEn mB8R1MKvYrmnYTPzNMk26lbJByeJngAZIbQ+x3SXVVsQOp1kUYuvDQivzNOKgSzS2Fk5dHQ8iguRf 0JRv07onWNjVa5ORY11q3sgv7cjIsI/nzzEHzFcjzgT8oPsexNajzCPyVVAAsypE9MOe8Qx39LZKG uGnzvRWQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76b-0003Xy-5H; Wed, 10 Jun 2020 20:13:49 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 46/51] mm: Add DEFINE_READAHEAD Date: Wed, 10 Jun 2020 13:13:40 -0700 Message-Id: <20200610201345.13273-47-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8796F1801BF71 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Allow for a more concise definition of a struct readahead_control. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/pagemap.h | 7 +++++++ mm/readahead.c | 6 +----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 8455a3e16900..77c20da5ae99 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -759,6 +759,13 @@ struct readahead_control { unsigned int _batch_count; }; +#define DEFINE_READAHEAD(rac, f, m, i) \ + struct readahead_control rac = { \ + .file = f, \ + .mapping = m, \ + ._index = i, \ + } + /** * readahead_page - Get the next page to read. * @rac: The current readahead request. diff --git a/mm/readahead.c b/mm/readahead.c index 3c9a8dd7c56c..2126a2754e22 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -179,11 +179,7 @@ void page_cache_readahead_unbounded(struct address_space *mapping, { LIST_HEAD(page_pool); gfp_t gfp_mask = readahead_gfp_mask(mapping); - struct readahead_control rac = { - .mapping = mapping, - .file = file, - ._index = index, - }; + DEFINE_READAHEAD(rac, file, mapping, index); unsigned long i; /* From patchwork Wed Jun 10 20:13:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598815 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C514190 for ; Wed, 10 Jun 2020 20:39:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 84B0A207ED for ; Wed, 10 Jun 2020 20:39:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="gzslWAeU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 84B0A207ED Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D388F8D001B; Wed, 10 Jun 2020 16:39:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CC0D98D000E; Wed, 10 Jun 2020 16:39:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC69D8D001B; Wed, 10 Jun 2020 16:39:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0149.hostedemail.com [216.40.44.149]) by kanga.kvack.org (Postfix) with ESMTP id 8B12F8D000E for ; Wed, 10 Jun 2020 16:39:06 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 54921CF9C4 for ; Wed, 10 Jun 2020 20:39:06 +0000 (UTC) X-FDA: 76914466692.10.smell33_4f030d026dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 295771AEC5 for ; Wed, 10 Jun 2020 20:39:06 +0000 (UTC) X-Spam-Summary: 2,0,0,87a1b10e96358087,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:2:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:2693:2898:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:3874:4049:4119:4321:4605:5007:6261:6653:7576:7875:7903:8957:9036:10004:11026:11232:11473:11658:11914:12043:12114:12296:12297:12438:12555:12895:12986:13255:13894:14096:14394:21080:21433:21451:21627:21990:30003:30034:30054:30070:30090,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:1,LUA_SUMMARY:none X-HE-Tag: smell33_4f030d026dce X-Filterd-Recvd-Size: 8038 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf23.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=idZKdA+QYZwMV+ZxoObzC46KdDw/yL2GIOrIWkTdIjI=; b=gzslWAeUZmmexSew2yt137OmdW SShvLXnxZK86Ft60vuDRgPt0P8y4I//kQLT5EG7MFMSQgA3iucbKNx2juCELExXL6vasD4C+IDXCh 3X6q8ZmXRGOfgaTTh/q7ltSxC5RQKBNf48oSlJEViO4tanp/94VI27uxgj3cgI2XQbn6+Bu1TNKv4 spLvOhlXgjdXxukuXE5Qyb/PnT6EOYftHFGn4VW5aYrSK0EREMv6eEp+U6s/Su88Ba+RgDBRkWwqr 4fgxb+B6dG8U/o0cmnzAmlTwAXDlaPtv4fKPh8GsRo2fFKGeWE/4gdQVFFmVrWHvJicmlL5sqTid3 slY27qOQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76b-0003Y5-6e; Wed, 10 Jun 2020 20:13:49 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 47/51] mm: Make page_cache_readahead_unbounded take a readahead_control Date: Wed, 10 Jun 2020 13:13:41 -0700 Message-Id: <20200610201345.13273-48-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 295771AEC5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Define it in the callers instead of in page_cache_readahead_unbounded(). Signed-off-by: Matthew Wilcox (Oracle) --- fs/ext4/verity.c | 4 ++-- fs/f2fs/verity.c | 4 ++-- include/linux/pagemap.h | 5 ++--- mm/readahead.c | 26 ++++++++++++-------------- 4 files changed, 18 insertions(+), 21 deletions(-) diff --git a/fs/ext4/verity.c b/fs/ext4/verity.c index dec1244dd062..fe2e541543da 100644 --- a/fs/ext4/verity.c +++ b/fs/ext4/verity.c @@ -346,6 +346,7 @@ static struct page *ext4_read_merkle_tree_page(struct inode *inode, pgoff_t index, unsigned long num_ra_pages) { + DEFINE_READAHEAD(rac, NULL, inode->i_mapping, index); struct page *page; index += ext4_verity_metadata_pos(inode) >> PAGE_SHIFT; @@ -355,8 +356,7 @@ static struct page *ext4_read_merkle_tree_page(struct inode *inode, if (page) put_page(page); else if (num_ra_pages > 1) - page_cache_readahead_unbounded(inode->i_mapping, NULL, - index, num_ra_pages, 0); + page_cache_readahead_unbounded(&rac, num_ra_pages, 0); page = read_mapping_page(inode->i_mapping, index, NULL); } return page; diff --git a/fs/f2fs/verity.c b/fs/f2fs/verity.c index 865c9fb774fb..707a94745472 100644 --- a/fs/f2fs/verity.c +++ b/fs/f2fs/verity.c @@ -226,6 +226,7 @@ static struct page *f2fs_read_merkle_tree_page(struct inode *inode, pgoff_t index, unsigned long num_ra_pages) { + DEFINE_READAHEAD(rac, NULL, inode->i_mapping, index); struct page *page; index += f2fs_verity_metadata_pos(inode) >> PAGE_SHIFT; @@ -235,8 +236,7 @@ static struct page *f2fs_read_merkle_tree_page(struct inode *inode, if (page) put_page(page); else if (num_ra_pages > 1) - page_cache_readahead_unbounded(inode->i_mapping, NULL, - index, num_ra_pages, 0); + page_cache_readahead_unbounded(&rac, num_ra_pages, 0); page = read_mapping_page(inode->i_mapping, index, NULL); } return page; diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 77c20da5ae99..f36714473b1e 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -715,9 +715,8 @@ void page_cache_sync_readahead(struct address_space *, struct file_ra_state *, void page_cache_async_readahead(struct address_space *, struct file_ra_state *, struct file *, struct page *, pgoff_t index, unsigned long req_count); -void page_cache_readahead_unbounded(struct address_space *, struct file *, - pgoff_t index, unsigned long nr_to_read, - unsigned long lookahead_count); +void page_cache_readahead_unbounded(struct readahead_control *, + unsigned long nr_to_read, unsigned long lookahead_count); /* * Like add_to_page_cache_locked, but used to add newly allocated pages: diff --git a/mm/readahead.c b/mm/readahead.c index 2126a2754e22..62da2d4beed1 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -159,9 +159,7 @@ static void read_pages(struct readahead_control *rac, struct list_head *pages, /** * page_cache_readahead_unbounded - Start unchecked readahead. - * @mapping: File address space. - * @file: This instance of the open file; used for authentication. - * @index: First page index to read. + * @rac: Readahead control. * @nr_to_read: The number of pages to read. * @lookahead_size: Where to start the next readahead. * @@ -173,13 +171,13 @@ static void read_pages(struct readahead_control *rac, struct list_head *pages, * Context: File is referenced by caller. Mutexes may be held by caller. * May sleep, but will not reenter filesystem to reclaim memory. */ -void page_cache_readahead_unbounded(struct address_space *mapping, - struct file *file, pgoff_t index, unsigned long nr_to_read, - unsigned long lookahead_size) +void page_cache_readahead_unbounded(struct readahead_control *rac, + unsigned long nr_to_read, unsigned long lookahead_size) { + struct address_space *mapping = rac->mapping; + unsigned long index = readahead_index(rac); LIST_HEAD(page_pool); gfp_t gfp_mask = readahead_gfp_mask(mapping); - DEFINE_READAHEAD(rac, file, mapping, index); unsigned long i; /* @@ -200,7 +198,7 @@ void page_cache_readahead_unbounded(struct address_space *mapping, for (i = 0; i < nr_to_read; i++) { struct page *page = xa_load(&mapping->i_pages, index + i); - BUG_ON(index + i != rac._index + rac._nr_pages); + BUG_ON(index + i != rac->_index + rac->_nr_pages); if (page && !xa_is_value(page)) { /* @@ -211,7 +209,7 @@ void page_cache_readahead_unbounded(struct address_space *mapping, * have a stable reference to this page, and it's * not worth getting one just for that. */ - read_pages(&rac, &page_pool, true); + read_pages(rac, &page_pool, true); continue; } @@ -224,12 +222,12 @@ void page_cache_readahead_unbounded(struct address_space *mapping, } else if (add_to_page_cache_lru(page, mapping, index + i, gfp_mask) < 0) { put_page(page); - read_pages(&rac, &page_pool, true); + read_pages(rac, &page_pool, true); continue; } if (i == nr_to_read - lookahead_size) SetPageReadahead(page); - rac._nr_pages++; + rac->_nr_pages++; } /* @@ -237,7 +235,7 @@ void page_cache_readahead_unbounded(struct address_space *mapping, * uptodate then the caller will launch readpage again, and * will then handle the error. */ - read_pages(&rac, &page_pool, false); + read_pages(rac, &page_pool, false); memalloc_nofs_restore(nofs); } EXPORT_SYMBOL_GPL(page_cache_readahead_unbounded); @@ -252,6 +250,7 @@ void __do_page_cache_readahead(struct address_space *mapping, struct file *file, pgoff_t index, unsigned long nr_to_read, unsigned long lookahead_size) { + DEFINE_READAHEAD(rac, file, mapping, index); struct inode *inode = mapping->host; loff_t isize = i_size_read(inode); pgoff_t end_index; /* The last page we want to read */ @@ -266,8 +265,7 @@ void __do_page_cache_readahead(struct address_space *mapping, if (nr_to_read > end_index - index) nr_to_read = end_index - index + 1; - page_cache_readahead_unbounded(mapping, file, index, nr_to_read, - lookahead_size); + page_cache_readahead_unbounded(&rac, nr_to_read, lookahead_size); } /* From patchwork Wed Jun 10 20:13:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598813 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CBB08618 for ; Wed, 10 Jun 2020 20:39:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 98503206A4 for ; Wed, 10 Jun 2020 20:39:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="tA0NucqI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 98503206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4AB068D001A; Wed, 10 Jun 2020 16:39:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3E6E38D000E; Wed, 10 Jun 2020 16:39:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B1F78D001A; Wed, 10 Jun 2020 16:39:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0095.hostedemail.com [216.40.44.95]) by kanga.kvack.org (Postfix) with ESMTP id 09ABF8D000E for ; Wed, 10 Jun 2020 16:39:05 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id C451E18001C0A for ; Wed, 10 Jun 2020 20:39:04 +0000 (UTC) X-FDA: 76914466608.22.cloth86_060a20026dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id 930251817196F for ; Wed, 10 Jun 2020 20:39:04 +0000 (UTC) X-Spam-Summary: 73,5,0,b35de780d754a203,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1605:1711:1730:1747:1777:1792:2198:2199:2344:2393:2559:2562:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3874:4117:4250:4321:5007:6261:6653:7576:7875:8603:8660:10004:11026:11232:11473:11658:11914:12043:12114:12296:12297:12438:12555:12895:12986:13148:13230:13894:14096:14181:14394:14721:21080:21433:21451:21627:21939:21990:30045:30054:30070,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: cloth86_060a20026dce X-Filterd-Recvd-Size: 6379 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=S8iTV89UZwBhFOKRSSgrXr6LwEGxyfVVQ8jpCAk+4T4=; b=tA0NucqILgtgEHPhR0QOz7pyrc zugbZGH0I3XvQq+j0ODT3muFUniByXxVfJLSRxJcvPAMz9m2bjaC/fRN1tNNET95jJMN4l1Jo6EbP zDUoM8IVmQhJ8A3sy7ZpcUR6deIKOxefOhQ6r6hr9fab4ulNboCcNY90y3WTGBHhF8yT1c5mKx2wo ZWpq0/whGdf/om/F+WMT5eITvlH4eqHcFu68r/6G+X+zyCg67O1AotBhoPTsfDvXb5DQ8N+4E+WvV skfW41Kd0LZiexsBMp+ievdcW71GYsTQb2ECzNtxJdb89Jp2D9OxUS9sz65vQx8W9g1uqSkE/i+Yv c6gTmipg==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76b-0003YF-8X; Wed, 10 Jun 2020 20:13:49 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 48/51] mm: Make __do_page_cache_readahead take a readahead_control Date: Wed, 10 Jun 2020 13:13:42 -0700 Message-Id: <20200610201345.13273-49-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 930251817196F X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Also call __do_page_cache_readahead() directly from ondemand_readahead() instead of indirecting via ra_submit(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/internal.h | 11 +++++------ mm/readahead.c | 26 ++++++++++++++------------ 2 files changed, 19 insertions(+), 18 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index cd7038a36354..63c493f892e2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -51,18 +51,17 @@ void unmap_page_range(struct mmu_gather *tlb, void force_page_cache_readahead(struct address_space *, struct file *, pgoff_t index, unsigned long nr_to_read); -void __do_page_cache_readahead(struct address_space *, struct file *, - pgoff_t index, unsigned long nr_to_read, - unsigned long lookahead_size); +void __do_page_cache_readahead(struct readahead_control *, + unsigned long nr_to_read, unsigned long lookahead_size); /* * Submit IO for the read-ahead request in file_ra_state. */ static inline void ra_submit(struct file_ra_state *ra, - struct address_space *mapping, struct file *filp) + struct address_space *mapping, struct file *file) { - __do_page_cache_readahead(mapping, filp, - ra->start, ra->size, ra->async_size); + DEFINE_READAHEAD(rac, file, mapping, ra->start); + __do_page_cache_readahead(&rac, ra->size, ra->async_size); } /** diff --git a/mm/readahead.c b/mm/readahead.c index 62da2d4beed1..74c7e1eff540 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -246,12 +246,11 @@ EXPORT_SYMBOL_GPL(page_cache_readahead_unbounded); * behaviour which would occur if page allocations are causing VM writeback. * We really don't want to intermingle reads and writes like that. */ -void __do_page_cache_readahead(struct address_space *mapping, - struct file *file, pgoff_t index, unsigned long nr_to_read, - unsigned long lookahead_size) +void __do_page_cache_readahead(struct readahead_control *rac, + unsigned long nr_to_read, unsigned long lookahead_size) { - DEFINE_READAHEAD(rac, file, mapping, index); - struct inode *inode = mapping->host; + struct inode *inode = rac->mapping->host; + unsigned long index = readahead_index(rac); loff_t isize = i_size_read(inode); pgoff_t end_index; /* The last page we want to read */ @@ -265,7 +264,7 @@ void __do_page_cache_readahead(struct address_space *mapping, if (nr_to_read > end_index - index) nr_to_read = end_index - index + 1; - page_cache_readahead_unbounded(&rac, nr_to_read, lookahead_size); + page_cache_readahead_unbounded(rac, nr_to_read, lookahead_size); } /* @@ -273,10 +272,11 @@ void __do_page_cache_readahead(struct address_space *mapping, * memory at once. */ void force_page_cache_readahead(struct address_space *mapping, - struct file *filp, pgoff_t index, unsigned long nr_to_read) + struct file *file, pgoff_t index, unsigned long nr_to_read) { + DEFINE_READAHEAD(rac, file, mapping, index); struct backing_dev_info *bdi = inode_to_bdi(mapping->host); - struct file_ra_state *ra = &filp->f_ra; + struct file_ra_state *ra = &file->f_ra; unsigned long max_pages; if (unlikely(!mapping->a_ops->readpage && !mapping->a_ops->readpages && @@ -294,7 +294,7 @@ void force_page_cache_readahead(struct address_space *mapping, if (this_chunk > nr_to_read) this_chunk = nr_to_read; - __do_page_cache_readahead(mapping, filp, index, this_chunk, 0); + __do_page_cache_readahead(&rac, this_chunk, 0); index += this_chunk; nr_to_read -= this_chunk; @@ -432,10 +432,11 @@ static int try_context_readahead(struct address_space *mapping, * A minimal readahead algorithm for trivial sequential/random reads. */ static void ondemand_readahead(struct address_space *mapping, - struct file_ra_state *ra, struct file *filp, + struct file_ra_state *ra, struct file *file, bool hit_readahead_marker, pgoff_t index, unsigned long req_size) { + DEFINE_READAHEAD(rac, file, mapping, index); struct backing_dev_info *bdi = inode_to_bdi(mapping->host); unsigned long max_pages = ra->ra_pages; unsigned long add_pages; @@ -516,7 +517,7 @@ static void ondemand_readahead(struct address_space *mapping, * standalone, small random read * Read as is, and do not pollute the readahead state. */ - __do_page_cache_readahead(mapping, filp, index, req_size, 0); + __do_page_cache_readahead(&rac, req_size, 0); return; initial_readahead: @@ -542,7 +543,8 @@ static void ondemand_readahead(struct address_space *mapping, } } - ra_submit(ra, mapping, filp); + rac._index = ra->start; + __do_page_cache_readahead(&rac, ra->size, ra->async_size); } /** From patchwork Wed Jun 10 20:13:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598821 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C211090 for ; Wed, 10 Jun 2020 20:39:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 82D17206A4 for ; Wed, 10 Jun 2020 20:39:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Ls7VeNeX" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 82D17206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 23ABF8D001E; Wed, 10 Jun 2020 16:39:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1C3348D000E; Wed, 10 Jun 2020 16:39:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 089458D001E; Wed, 10 Jun 2020 16:39:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0084.hostedemail.com [216.40.44.84]) by kanga.kvack.org (Postfix) with ESMTP id E37DA8D000E for ; Wed, 10 Jun 2020 16:39:13 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A7E8D801AC79 for ; Wed, 10 Jun 2020 20:39:13 +0000 (UTC) X-FDA: 76914466986.30.head61_051604b26dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 6C35618022685 for ; Wed, 10 Jun 2020 20:39:13 +0000 (UTC) X-Spam-Summary: 2,0,0,b4fdc52adf6eda7a,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1542:1711:1730:1747:1777:1792:2194:2198:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3353:3865:3867:3871:4321:4605:5007:6261:6653:7576:7875:8660:8957:10004:10226:11026:11232:11473:11658:11914:12043:12296:12297:12438:12555:12895:12986:13148:13230:13894:14096:14181:14394:14721:21080:21324:21451:21627:21939:21990:30054,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-62.8.0.100 64.201.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:15,LUA_SUMMARY:none X-HE-Tag: head61_051604b26dce X-Filterd-Recvd-Size: 3984 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=2hV4r5m+TkSlkdyqQBMdETJhjIoOoHgfZkbofMON82Y=; b=Ls7VeNeXB1UfVNMNYLfV2oi1dO T91HRkOKlYRXqP9eRJ1Eiz1pGfQSUAGume/MeHGML1a1tNdUCrBeWIt430Gw7vWFyOeTvg/y1BkEe 2iCIPcIeuwjLlYAo4hAc+u4Vmf18qTQKrZBkFkHT+2pWWp2tU+JZ26wh8m7dQJZHzOor6klOEldC+ XEGnpxT4gwO+XkJ7ITi1Uj9ym/NmAsKDq8t70DSz0M5Xth9HqUbiBl2j8SLgI7eEckUW0KP1XWj/J uQeOL2JIIxPCDH9V3lEwS5S0s6HbHOWdpO5khzTEj11V9jxKR1xpfo7AqsAeO+VwvHz5KYXxUYM9f g0ROXf+Q==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76b-0003YM-9t; Wed, 10 Jun 2020 20:13:49 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 49/51] mm: Allow PageReadahead to be set on head pages Date: Wed, 10 Jun 2020 13:13:43 -0700 Message-Id: <20200610201345.13273-50-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6C35618022685 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" Adjust the callers to only call PageReadahead on the head page. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/page-flags.h | 4 ++-- mm/filemap.c | 14 +++++++------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 979460df4768..a3110d675cd0 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -377,8 +377,8 @@ PAGEFLAG(MappedToDisk, mappedtodisk, PF_NO_TAIL) /* PG_readahead is only used for reads; PG_reclaim is only for writes */ PAGEFLAG(Reclaim, reclaim, PF_NO_TAIL) TESTCLEARFLAG(Reclaim, reclaim, PF_NO_TAIL) -PAGEFLAG(Readahead, reclaim, PF_NO_COMPOUND) - TESTCLEARFLAG(Readahead, reclaim, PF_NO_COMPOUND) +PAGEFLAG(Readahead, reclaim, PF_ONLY_HEAD) + TESTCLEARFLAG(Readahead, reclaim, PF_ONLY_HEAD) #ifdef CONFIG_HIGHMEM /* diff --git a/mm/filemap.c b/mm/filemap.c index 869b970fe1ab..7a746b486237 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2064,9 +2064,9 @@ ssize_t generic_file_buffered_read(struct kiocb *iocb, if (unlikely(page == NULL)) goto no_cached_page; } - if (PageReadahead(page)) { + if (PageReadahead(thp_head(page))) { page_cache_async_readahead(mapping, - ra, filp, page, + ra, filp, thp_head(page), index, last_index - index); } if (!PageUptodate(page)) { @@ -2452,10 +2452,10 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf, return fpin; if (ra->mmap_miss > 0) ra->mmap_miss--; - if (PageReadahead(page)) { + if (PageReadahead(thp_head(page))) { fpin = maybe_unlock_mmap_for_io(vmf, fpin); page_cache_async_readahead(mapping, ra, file, - page, offset, ra->ra_pages); + thp_head(page), offset, ra->ra_pages); } return fpin; } @@ -2637,11 +2637,11 @@ void filemap_map_pages(struct vm_fault *vmf, /* Has the page moved or been split? */ if (unlikely(page != xas_reload(&xas))) goto skip; + if (PageReadahead(page)) + goto skip; page = find_subpage(page, xas.xa_index); - if (!PageUptodate(page) || - PageReadahead(page) || - PageHWPoison(page)) + if (!PageUptodate(page) || PageHWPoison(page)) goto skip; if (!trylock_page(page)) goto skip; From patchwork Wed Jun 10 20:13:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598809 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1E192618 for ; Wed, 10 Jun 2020 20:39:05 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DF2A6207ED for ; Wed, 10 Jun 2020 20:39:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="VD7fU7Lu" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DF2A6207ED Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 944158D0018; Wed, 10 Jun 2020 16:39:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8C9CA8D000E; Wed, 10 Jun 2020 16:39:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 748258D0018; Wed, 10 Jun 2020 16:39:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0103.hostedemail.com [216.40.44.103]) by kanga.kvack.org (Postfix) with ESMTP id 4E8B48D000E for ; Wed, 10 Jun 2020 16:39:02 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0C5AA80354E4 for ; Wed, 10 Jun 2020 20:39:02 +0000 (UTC) X-FDA: 76914466524.20.maid95_6302c3126dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id DD0FE180C10ED for ; Wed, 10 Jun 2020 20:39:01 +0000 (UTC) X-Spam-Summary: 2,0,0,dc2f0fc9780d57ec,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:421:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1544:1605:1711:1730:1747:1777:1792:2198:2199:2393:2553:2559:2562:2693:2731:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3872:4117:4250:4321:5007:6119:6120:6261:6653:7576:7875:7901:7903:8603:8957:9036:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12555:12683:12895:13894:14096:14110:14181:14394:14721:21080:21433:21451:21627:21990:30012:30054:30056:30090,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:22,LUA_SUMMARY:none X-HE-Tag: maid95_6302c3126dce X-Filterd-Recvd-Size: 6402 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=MzsyfRnyDe67a2Cfz5uY3LqHozA0IstJ5c01vhKwG9I=; b=VD7fU7LuZ61xGvb1zRlmkXDnh4 +J3601D7omZ47AOcyP+VEXSGypqy+y0+hH94n8UEFLeQPDc9dsY5Bu2qWFAiwrAzglRamvVcwu+7e d848ODtU2nVCwl6T6jTIu7CidmoIFW1nRasfwKVUVqR6Gbo9E6hfsX6u+0FHkIXuquVSbZsrp7L6s QunVJjJ+xkv3jR36/3bmAKO2y7f9H7k/36I5mAdyPKpS/Z+vlPZca6tXIviFNHXpLj78UH1o4bpk/ +cTr/CPFU8TDrENMB7Y1x9EXwm/3E5+x6cjoODs5Zjc2SxJhs/ZkfbNLuM/fmRCDAwnGwV1jLpy6W KRUk9+Pg==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76b-0003YT-B2; Wed, 10 Jun 2020 20:13:49 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 50/51] mm: Add THP readahead Date: Wed, 10 Jun 2020 13:13:44 -0700 Message-Id: <20200610201345.13273-51-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: DD0FE180C10ED X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" If the filesystem supports THPs, allocate larger pages in the readahead code when it seems worth doing. The heuristic for choosing larger page sizes will surely need some tuning, but this aggressive ramp-up seems good for testing. Signed-off-by: Matthew Wilcox (Oracle) --- mm/readahead.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 87 insertions(+), 6 deletions(-) diff --git a/mm/readahead.c b/mm/readahead.c index 74c7e1eff540..98bbcc986b39 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -149,7 +149,7 @@ static void read_pages(struct readahead_control *rac, struct list_head *pages, blk_finish_plug(&plug); - BUG_ON(!list_empty(pages)); + BUG_ON(pages && !list_empty(pages)); BUG_ON(readahead_count(rac)); out: @@ -428,13 +428,92 @@ static int try_context_readahead(struct address_space *mapping, return 1; } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static inline int ra_alloc_page(struct readahead_control *rac, pgoff_t index, + pgoff_t mark, unsigned int order, gfp_t gfp) +{ + int err; + struct page *page = __page_cache_alloc_order(gfp, order); + + if (!page) + return -ENOMEM; + if (mark - index < (1UL << order)) + SetPageReadahead(page); + err = add_to_page_cache_lru(page, rac->mapping, index, gfp); + if (err) + put_page(page); + else + rac->_nr_pages += 1UL << order; + return err; +} + +static bool page_cache_readahead_order(struct readahead_control *rac, + struct file_ra_state *ra, unsigned int order) +{ + struct address_space *mapping = rac->mapping; + unsigned int old_order = order; + pgoff_t index = readahead_index(rac); + pgoff_t limit = (i_size_read(mapping->host) - 1) >> PAGE_SHIFT; + pgoff_t mark = index + ra->size - ra->async_size; + int err = 0; + gfp_t gfp = readahead_gfp_mask(mapping); + + if (!mapping_thp_support(mapping)) + return false; + + limit = min(limit, index + ra->size - 1); + + /* Grow page size up to PMD size */ + if (order < HPAGE_PMD_ORDER) { + order += 2; + if (order > HPAGE_PMD_ORDER) + order = HPAGE_PMD_ORDER; + while ((1 << order) > ra->size) + order--; + } + + /* If size is somehow misaligned, fill with order-0 pages */ + while (!err && index & ((1UL << old_order) - 1)) + err = ra_alloc_page(rac, index++, mark, 0, gfp); + + while (!err && index & ((1UL << order) - 1)) { + err = ra_alloc_page(rac, index, mark, old_order, gfp); + index += 1UL << old_order; + } + + while (!err && index <= limit) { + err = ra_alloc_page(rac, index, mark, order, gfp); + index += 1UL << order; + } + + if (index > limit) { + ra->size += index - limit - 1; + ra->async_size += index - limit - 1; + } + + read_pages(rac, NULL, false); + + /* + * If there were already pages in the page cache, then we may have + * left some gaps. Let the regular readahead code take care of this + * situation. + */ + return !err; +} +#else +static bool page_cache_readahead_order(struct readahead_control *rac, + struct file_ra_state *ra, unsigned int order) +{ + return false; +} +#endif + /* * A minimal readahead algorithm for trivial sequential/random reads. */ static void ondemand_readahead(struct address_space *mapping, struct file_ra_state *ra, struct file *file, - bool hit_readahead_marker, pgoff_t index, - unsigned long req_size) + struct page *page, pgoff_t index, unsigned long req_size) { DEFINE_READAHEAD(rac, file, mapping, index); struct backing_dev_info *bdi = inode_to_bdi(mapping->host); @@ -473,7 +552,7 @@ static void ondemand_readahead(struct address_space *mapping, * Query the pagecache for async_size, which normally equals to * readahead size. Ramp it up and use it as the new readahead size. */ - if (hit_readahead_marker) { + if (page) { pgoff_t start; rcu_read_lock(); @@ -544,6 +623,8 @@ static void ondemand_readahead(struct address_space *mapping, } rac._index = ra->start; + if (page && page_cache_readahead_order(&rac, ra, thp_order(page))) + return; __do_page_cache_readahead(&rac, ra->size, ra->async_size); } @@ -578,7 +659,7 @@ void page_cache_sync_readahead(struct address_space *mapping, } /* do read-ahead */ - ondemand_readahead(mapping, ra, filp, false, index, req_count); + ondemand_readahead(mapping, ra, filp, NULL, index, req_count); } EXPORT_SYMBOL_GPL(page_cache_sync_readahead); @@ -624,7 +705,7 @@ page_cache_async_readahead(struct address_space *mapping, return; /* do read-ahead */ - ondemand_readahead(mapping, ra, filp, true, index, req_count); + ondemand_readahead(mapping, ra, filp, page, index, req_count); } EXPORT_SYMBOL_GPL(page_cache_async_readahead); From patchwork Wed Jun 10 20:13:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 11598817 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BEDC9618 for ; Wed, 10 Jun 2020 20:39:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 81885206A4 for ; Wed, 10 Jun 2020 20:39:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="k38Y8vKb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 81885206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 325588D001C; Wed, 10 Jun 2020 16:39:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2AF1C8D000E; Wed, 10 Jun 2020 16:39:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17A978D001C; Wed, 10 Jun 2020 16:39:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DB3218D000E for ; Wed, 10 Jun 2020 16:39:07 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9BABE11F695 for ; Wed, 10 Jun 2020 20:39:07 +0000 (UTC) X-FDA: 76914466734.16.women86_391832526dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 732AE100EEF15 for ; Wed, 10 Jun 2020 20:39:07 +0000 (UTC) X-Spam-Summary: 2,0,0,f055578ea819951d,d41d8cd98f00b204,willy@infradead.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3867:4031:5007:6120:6261:6653:7576:7901:9592:10004:11026:11473:11658:11914:12043:12297:12438:12555:12679:12895:13069:13184:13229:13311:13357:13894:14096:14181:14384:14394:14721:21080:21451:21627:21740:21939:21990:30054:30064,0,RBL:198.137.202.133:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: women86_391832526dce X-Filterd-Recvd-Size: 2653 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=Syv05ntpXp5OHYqqfiVzAsVsVw2vQPR8ml3h23JrCRo=; b=k38Y8vKbdkeU0De3i2bolBbeNS SSCL2Q6QOBrIPOIJD3gJVAC1Xhfp5AlLeatX62Mp4ZTvK3R3F/kg/IZJGEXy5aPbOeq6qIBuVyFtN eyVFm0PEI/I6PglMho1hMAsthcYPja6a+C17ZkEompoDPvse2TEvyBRMt0ZV1oellCcxXGsM0xjwD tEn/46jEJQxhV9Xla/3bnL65pnz17bIyRgwGG7+s0GnrsODzEiUSZtsW9XwNRtSF4IK7cTpMAAvuG Piu1/kELPZEPiwa3JpdGmZhv1iItnVAh2yu+/CvEFmUSzFtvP9pMFR97ViA07JPXTwbDunyDeUxtH UgiVNH1Q==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76b-0003YY-CU; Wed, 10 Jun 2020 20:13:49 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: William Kucharski , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Matthew Wilcox Subject: [PATCH v6 51/51] mm: Align THP mappings for non-DAX Date: Wed, 10 Jun 2020 13:13:45 -0700 Message-Id: <20200610201345.13273-52-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 732AE100EEF15 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: William Kucharski When we have the opportunity to use transparent huge pages to map a file, we want to follow the same rules as DAX. Signed-off-by: William Kucharski Signed-off-by: Matthew Wilcox (Oracle) --- mm/huge_memory.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c25d8e2310e8..0fb08e4eeb8e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -577,13 +577,10 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr, unsigned long ret; loff_t off = (loff_t)pgoff << PAGE_SHIFT; - if (!IS_DAX(filp->f_mapping->host) || !IS_ENABLED(CONFIG_FS_DAX_PMD)) - goto out; - ret = __thp_get_unmapped_area(filp, addr, len, off, flags, PMD_SIZE); if (ret) return ret; -out: + return current->mm->get_unmapped_area(filp, addr, len, pgoff, flags); } EXPORT_SYMBOL_GPL(thp_get_unmapped_area);