From patchwork Tue Feb 7 19:49:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13132076 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D6ABC636D3 for ; Tue, 7 Feb 2023 19:49:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA49A6B0111; Tue, 7 Feb 2023 14:49:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D02F26B010F; Tue, 7 Feb 2023 14:49:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7C7B900002; Tue, 7 Feb 2023 14:49:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A64946B010E for ; Tue, 7 Feb 2023 14:49:46 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7BB6BAB17A for ; Tue, 7 Feb 2023 19:49:46 +0000 (UTC) X-FDA: 80441535972.12.5C0D0D3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf01.hostedemail.com (Postfix) with ESMTP id 6761F40008 for ; Tue, 7 Feb 2023 19:49:44 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jiDG9UC6; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675799384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RwrLOQlQFZldBSoSj0NqBnJnjxxeVCBmAtcieBK8nys=; b=PydsKObeRIhuJmfb9p03M4pH3E26SUkAe6w2On6ERjso1iVVOwWPdQAwU9XggWlctulAzr WlxX5RRVeJa2eLVpwnfxZVqA9gRiK1mLliUEhUonF87IaIISP1fE9uC9DVuXysyt00mQNd kpDZdX1NkGS/mQwaCndgxZqtaHOpKDs= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jiDG9UC6; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675799384; a=rsa-sha256; cv=none; b=VMreUm+RbDpIK/C2oFjEQCyFWQKPshV1msQBGfiQeLks7m7moy5G6niERdaHxoY1zvoFhu ++73yp6a+7s1X61ckIOpERuHCMApv6pf46lvh5tRyy/XUiZPILSWl0K1Omilh9gYdZ0hQN gfeN5zTDysujYh3KsnM7SYRp3xgxiPY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=RwrLOQlQFZldBSoSj0NqBnJnjxxeVCBmAtcieBK8nys=; b=jiDG9UC64itoB7qW+fvXO/P9Wi s6hd3Sq9wsR1k4OFUeEwoRq/en5nDc7lkB2dz9F8wUtHky8pEJy2PgeEvNU1lu5DzNVwMnXsmmwhg hLiPo8FksL+3Icm1Q+nz1Fdn/i4zIQmuQ7hh5BoFwdRzxAA+AOh5c5SGEBK/k93VXYBqgRuTttVSD hXCLD7C+M2tg9Kz4lN0CL4Gi6JC6WbOqzW7AVHRlpvvGPLnIlUQuOBFcBlLyOV/93qtdWtbDWAMK7 a0UgtF/tFtdcqcmmIVP/6IQh2D3jwUC+a2zTklVhyb18d3qgIezijrnqWVv8YsPKH/Bf+THlXtEFk ECsHt78w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pPTyH-000Vt1-8N; Tue, 07 Feb 2023 19:49:41 +0000 From: "Matthew Wilcox (Oracle)" To: Yin Fengwei , linux-mm@kvack.org Cc: david@redhat.com, dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, Matthew Wilcox Subject: [PATCH v5 5/5] filemap: Batch PTE mappings Date: Tue, 7 Feb 2023 19:49:35 +0000 Message-Id: <20230207194937.122543-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230207194937.122543-1-willy@infradead.org> References: <20230207194937.122543-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6761F40008 X-Stat-Signature: caibjnsbo9iy4xsxo13tn18grygajwwc X-HE-Tag: 1675799384-230873 X-HE-Meta: U2FsdGVkX1/VNGjwl1fvnY8eH0jqQokGYsZ0UXEOlkHUR3Ls1VrVxjpbW8/jH0tBS0+m9QVypzkbyITD8m3pQSww15MOM+tfxIm1mCJlSq4GG8IzVKKcfqFH8Q01ho8WwBlwDgwhZOc34ldsLpvSReO4BlY4konhrDHF0o5gFmjtGaygsHgC2cxOQa1twgbDMiDZ3atuwNoCnGUB4GNnAAWLbe89pv4LQHm7hKIUPOcXxUtKhLU3pZLepQunPKXW28OpIEPI9QSEHoRceW2DhsB0Ve879JdggNp5wOdKC8OYBRGfQXTZ5lpsUrHbd0o+wfnf4zGR3EibjZ8einbitZcKK34jpbTGwizVKhGf19AcBEGdRNMPVBmokFYP6FpL+EIz6KgpVrWpnI++J2a6vUW+bhUxy8Fg8lyBGFhpPkteIvFtQad3B2z6+4ZuzGk+HKzHzYb8aOWVUenDZPG6DJ4LJHTQfCxgO6m3LHcGQH5A7uYUttg5bq/kuF22MShHcg56IIc1BaWOgoPgE8ABPaq7OW7KfdMXNO8vVKXWmVd281Ij8q7mN6KjQ8HGQ/be2ulKHRzrNtIUJXzo8GBlZRrgZC02n6Oerk+1iutNUEjwYrnW0nJggmhn4oGoPsExeqAAf4jZGwzgQ50PiVH2fkn56iYDee8Bm4RmidV/f5r5D74XJ84zAk8mGnVd7n7RBJjwNKbSJWZ9h76mzQRRGB4sDRUl3QC62pW2lH3qKRtVVfCv/q1HfHzyTE1UHE3ANW4ffVlDL0TmIkPh+BO0zOxZzs1ms5cteIxTSDBiLuzb63HLcbTWWfbBcM7V9nIL3oVd6aSECZAZXG5efE0fzlbdefa/PEuMowt+BKRulYwNfTG/GB7mXN0Cu2l094BQnIoj+okoa7ADxsko9OPxu49CXkPB6WuY/xHB24HpqdPV9TEk4uaGOS6UY/yy4MQuM/zkQ0blUjms6rORF8/ lUZmaOzG 4FBRZ9Feqpy85T/6A4Tb8scS147svMwaeozjMjWC5h6EBviFBuAZm+3BXzmsxr1ZMs4oxKbSSho53uIK31QCradDOLJKri7foMcK9KXDGFB0AjoHrOP+hYy07L7uU6rX1UdZ46Q+gxAH6JLH6h6etfnbPYKrjThbYyUrAP1f1kXt4mMtt7pkwVgIHiOrapaEXnzIqo//RlcNZY3DDdsDqYckHRY9PXg+P+xDcQ9W0PAnMjAfflSY0+LUDug== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yin Fengwei Call set_pte_range() once per contiguous range of the folio instead of once per page. This batches the updates to mm counters and the rmap. With a will-it-scale.page_fault3 like app (change file write fault testing to read fault testing. Trying to upstream it to will-it-scale at [1]) got 15% performance gain on a 48C/96T Cascade Lake test box with 96 processes running against xfs. Perf data collected before/after the change: 18.73%--page_add_file_rmap | --11.60%--__mod_lruvec_page_state | |--7.40%--__mod_memcg_lruvec_state | | | --5.58%--cgroup_rstat_updated | --2.53%--__mod_lruvec_state | --1.48%--__mod_node_page_state 9.93%--page_add_file_rmap_range | --2.67%--__mod_lruvec_page_state | |--1.95%--__mod_memcg_lruvec_state | | | --1.57%--cgroup_rstat_updated | --0.61%--__mod_lruvec_state | --0.54%--__mod_node_page_state The running time of __mode_lruvec_page_state() is reduced about 9%. [1]: https://github.com/antonblanchard/will-it-scale/pull/37 Signed-off-by: Yin Fengwei Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 36 +++++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 745f1eb2a87f..4abab43d9564 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3354,11 +3354,12 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, struct file *file = vma->vm_file; struct page *page = folio_page(folio, start); unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); - unsigned int ref_count = 0, count = 0; + unsigned int count = 0; + pte_t *old_ptep = vmf->pte; do { - if (PageHWPoison(page)) - continue; + if (PageHWPoison(page + count)) + goto skip; if (mmap_miss > 0) mmap_miss--; @@ -3368,20 +3369,33 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, * handled in the specific fault path, and it'll prohibit the * fault-around logic. */ - if (!pte_none(*vmf->pte)) - continue; + if (!pte_none(vmf->pte[count])) + goto skip; if (vmf->address == addr) ret = VM_FAULT_NOPAGE; - ref_count++; - set_pte_range(vmf, folio, page, 1, addr); - } while (vmf->pte++, page++, addr += PAGE_SIZE, ++count < nr_pages); + count++; + continue; +skip: + if (count) { + set_pte_range(vmf, folio, page, count, addr); + folio_ref_add(folio, count); + } - /* Restore the vmf->pte */ - vmf->pte -= nr_pages; + count++; + page += count; + vmf->pte += count; + addr += count * PAGE_SIZE; + count = 0; + } while (--nr_pages > 0); + + if (count) { + set_pte_range(vmf, folio, page, count, addr); + folio_ref_add(folio, count); + } - folio_ref_add(folio, ref_count); + vmf->pte = old_ptep; WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss); return ret;