From patchwork Tue Feb 7 19:49:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13132080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B55BCC636D4 for ; Tue, 7 Feb 2023 19:49:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2085A6B0113; Tue, 7 Feb 2023 14:49:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1B84D6B0116; Tue, 7 Feb 2023 14:49:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A7B46B0118; Tue, 7 Feb 2023 14:49:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F02356B0113 for ; Tue, 7 Feb 2023 14:49:56 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 80C3C80B1D for ; Tue, 7 Feb 2023 19:49:56 +0000 (UTC) X-FDA: 80441536392.02.0F8FF6A Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf07.hostedemail.com (Postfix) with ESMTP id CFEBA4001C for ; Tue, 7 Feb 2023 19:49:54 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=HjrvvkkO; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675799394; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bn5An6YEWxMcfUEOmywANJ1SZY1Ff2p38vPeWaR0GaE=; b=dlw85Ao5GxIDXj8mgODHAOjkJhA8AP0dQnkyBm5ozdbe/hGEXWtl5ld5Nkqu/KOPfAcZKV OCLyrQzHUQXn0hTPO7PLbHzB7IWZxabc2dAsy23SxE8BkTADxdmyaQjyT6/aKSBa5NN/Fq s/Whm3rTQnjrRWLQqWgJhxZBuAgy2Gs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=HjrvvkkO; spf=none (imf07.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675799394; a=rsa-sha256; cv=none; b=pQXalnlEhpIlcWNNx+QzgW+yv2dyhLVohkWf7brXqm2bycsfN9WOPrQbvCZkhEfQVV85LJ BSm5hL5zBAgIUDaHtM5kv0vUqSdnxYGD8c9lV41HK2jGVZweMkxq1JxJjFkdvfWffOgX87 Bh67y4/XEUPa84MxsK38jigjUSgurR0= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=bn5An6YEWxMcfUEOmywANJ1SZY1Ff2p38vPeWaR0GaE=; b=HjrvvkkO9s+2iI37Dc/8I1NP9R xT2giehYlbAF1vDzT9DzJOk4b69YJhBJI6yWGYOSLcfuEFug3o5XW4T8pJBv3iL1l4KNbx+Bjw2JO mylPqHi3UAlh/Mrz3BhlMJUZPZ3jieKSgSjY5equAqi90yKEiuFPT1shhqAm2f3OJ8aFGusDUyaGe 68CAlLq8y/GIfUzhdMZNkxBsGJqkyhYq+96g1MWWbDnWd0tLOCz93P5ugKH/T+pAyXgH26HIlEBs/ mhqjcs7lgXjT8Prnbm/F9o8o9JmQpBn2ZxH3M7igg8KcXwElERG6vnvvzVOE/e1WQdU8IxJIoMb9m ma0JoM3A==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pPTyF-000Vsp-KC; Tue, 07 Feb 2023 19:49:39 +0000 From: "Matthew Wilcox (Oracle)" To: Yin Fengwei , linux-mm@kvack.org Cc: david@redhat.com, dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, Matthew Wilcox Subject: [PATCH v5 1/5] filemap: Add filemap_map_folio_range() Date: Tue, 7 Feb 2023 19:49:31 +0000 Message-Id: <20230207194937.122543-2-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230207194937.122543-1-willy@infradead.org> References: <20230207194937.122543-1-willy@infradead.org> MIME-Version: 1.0 X-Stat-Signature: kqh485goj64i3jc6bjsqppud5mkzhz3p X-Rspam-User: X-Rspamd-Queue-Id: CFEBA4001C X-Rspamd-Server: rspam06 X-HE-Tag: 1675799394-555937 X-HE-Meta: U2FsdGVkX1/NoYGCMcREry7wW4A12kwQJyL2FtOMvLTQi2sY/1p7TuVbPMnENwHgIrS9nKtiuaOSVWJ5UOfzpw/o2+0IMDc3hBHsOg9UJPA4uyn4u35gvvtSp6qtHNuA2gJ728AW6vW6uzu4NeaONZ1rCYbFpV5Yri16+PstouZK3XBlPtIjdoqCzUQ249K5lxrfKbOr6lcRhUIsccGpcpQggTmxcfHWSPD8c2VE8op3WbdDxjUvRSqV8BBkMZYWogL1jjqv3up51FcHK8rwkWY+tAAjJf8zsznp5dhrL4NLHCr6XZ2VNBm5UwjPI3dCs/QrPDVBvrY7bT8dKDLwlSqYQgtfgx2/ZCrRxlEmqIWuIokOpbKQL+0C+htpLUZ5EC7plwW02fVjOZy8M7gdoIsEDm1VNt/iTAM6J3Gx43tUxOLU+nUkU2YiQKuSbmFv6Y7eXI1ZlJ5PQePaob9mn2XHQuBmqSedhAR9ql1TwqyGczoC4bbMl0tWKt62rM1tfP8uB8n8tJOt837U9ZC+JNyjyrR3l5bzw5xzKQ9uOh5Zb32dA673ihMyT233xqKNuuALYpBMyysEIAPJ6f82PWYut37fDzmvZOUDtisrBMavp+H/yr22rfkvTPgF7dnLLpgI8hR8Mh1C+RCIRrcyDmbMJvYobmE0uUz/ylujGen+g3xX+CZAhz5XWC3j2CpqKVH1V/EmNsoWqdJXEC/FNePg1gFP9uFj9RSy6YmG8HKUCaczGwAVYXnsZKtRdAESvXCF9EBFOquJz3oT4jA1lw8fWFj2LeiffgrtstP4Tt8+Eh0qJaqdtXxLaW+b+igcr4+BCaRd+HVMXFy2gupurDH0z0uRt3kyq5NJjmzCTA6PX2LDYG6t8Kf6XB7ia/YGSyVDUXTibxAimreEAf9N+VxHfqg1HxDkeX35Iz7pEvJkhEMt4RoGncR+fyQDPqybXHxUwOX0M327SyFsC+H D/5pXWbY V5udOpJVJQ9G42azfy7nePGv86Xe0Dv9CZ/dEDg7WIz20ddQAzDLOm6Pep9G7ozg3ZxLqjHNa/Cr5w+LRUI7nSWNWBVtC490MOguhgyHqAnmFgB3kymdfMzinxnpOGY3xbelyrsPySzG8gaK2aD/hDQM1yJDR848qtXk4Dxb9Ssadqv4wJMrdy1T8MS5Ju7vKC0qiTawk8vJk07w9oRaquMnd7eV0jIcj7xuFZDc8PmXphekv3TWPETVbK39/VkUBsayrl6GBxxtlwZ8z5yKs7XI5JVDTfRVGbZkz X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yin Fengwei filemap_map_folio_range() maps partial/full folio. Comparing to original filemap_map_pages(), it updates refcount once per folio instead of per page and gets minor performance improvement for large folio. With a will-it-scale.page_fault3 like app (change file write fault testing to read fault testing. Trying to upstream it to will-it-scale at [1]), got 2% performance gain on a 48C/96T Cascade Lake test box with 96 processes running against xfs. [1]: https://github.com/antonblanchard/will-it-scale/pull/37 Signed-off-by: Yin Fengwei Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 98 +++++++++++++++++++++++++++++----------------------- 1 file changed, 54 insertions(+), 44 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 2ebcf500871d..9aa7a4fdc374 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2200,16 +2200,6 @@ unsigned filemap_get_folios(struct address_space *mapping, pgoff_t *start, } EXPORT_SYMBOL(filemap_get_folios); -static inline -bool folio_more_pages(struct folio *folio, pgoff_t index, pgoff_t max) -{ - if (!folio_test_large(folio) || folio_test_hugetlb(folio)) - return false; - if (index >= max) - return false; - return index < folio->index + folio_nr_pages(folio) - 1; -} - /** * filemap_get_folios_contig - Get a batch of contiguous folios * @mapping: The address_space to search @@ -3351,6 +3341,53 @@ static inline struct folio *next_map_page(struct address_space *mapping, mapping, xas, end_pgoff); } +/* + * Map page range [start_page, start_page + nr_pages) of folio. + * start_page is gotten from start by folio_page(folio, start) + */ +static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, + struct folio *folio, unsigned long start, + unsigned long addr, unsigned int nr_pages) +{ + vm_fault_t ret = 0; + struct vm_area_struct *vma = vmf->vma; + struct file *file = vma->vm_file; + struct page *page = folio_page(folio, start); + unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); + unsigned int ref_count = 0, count = 0; + + do { + if (PageHWPoison(page)) + continue; + + if (mmap_miss > 0) + mmap_miss--; + + /* + * NOTE: If there're PTE markers, we'll leave them to be + * handled in the specific fault path, and it'll prohibit the + * fault-around logic. + */ + if (!pte_none(*vmf->pte)) + continue; + + if (vmf->address == addr) + ret = VM_FAULT_NOPAGE; + + ref_count++; + do_set_pte(vmf, page, addr); + update_mmu_cache(vma, addr, vmf->pte); + } while (vmf->pte++, page++, addr += PAGE_SIZE, ++count < nr_pages); + + /* Restore the vmf->pte */ + vmf->pte -= nr_pages; + + folio_ref_add(folio, ref_count); + WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss); + + return ret; +} + vm_fault_t filemap_map_pages(struct vm_fault *vmf, pgoff_t start_pgoff, pgoff_t end_pgoff) { @@ -3361,9 +3398,9 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, unsigned long addr; XA_STATE(xas, &mapping->i_pages, start_pgoff); struct folio *folio; - struct page *page; unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); vm_fault_t ret = 0; + int nr_pages = 0; rcu_read_lock(); folio = first_map_page(mapping, &xas, end_pgoff); @@ -3378,45 +3415,18 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); do { -again: - page = folio_file_page(folio, xas.xa_index); - if (PageHWPoison(page)) - goto unlock; - - if (mmap_miss > 0) - mmap_miss--; + unsigned long end; addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT; vmf->pte += xas.xa_index - last_pgoff; last_pgoff = xas.xa_index; + end = folio->index + folio_nr_pages(folio) - 1; + nr_pages = min(end, end_pgoff) - xas.xa_index + 1; - /* - * NOTE: If there're PTE markers, we'll leave them to be - * handled in the specific fault path, and it'll prohibit the - * fault-around logic. - */ - if (!pte_none(*vmf->pte)) - goto unlock; - - /* We're about to handle the fault */ - if (vmf->address == addr) - ret = VM_FAULT_NOPAGE; + ret |= filemap_map_folio_range(vmf, folio, + xas.xa_index - folio->index, addr, nr_pages); + xas.xa_index += nr_pages; - do_set_pte(vmf, page, addr); - /* no need to invalidate: a not-present page won't be cached */ - update_mmu_cache(vma, addr, vmf->pte); - if (folio_more_pages(folio, xas.xa_index, end_pgoff)) { - xas.xa_index++; - folio_ref_inc(folio); - goto again; - } - folio_unlock(folio); - continue; -unlock: - if (folio_more_pages(folio, xas.xa_index, end_pgoff)) { - xas.xa_index++; - goto again; - } folio_unlock(folio); folio_put(folio); } while ((folio = next_map_page(mapping, &xas, end_pgoff)) != NULL); From patchwork Tue Feb 7 19:49:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13132077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAF0BC636CC for ; Tue, 7 Feb 2023 19:49:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 087626B010E; Tue, 7 Feb 2023 14:49:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 031D06B0112; Tue, 7 Feb 2023 14:49:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA0126B010E; Tue, 7 Feb 2023 14:49:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BADCF90000A for ; Tue, 7 Feb 2023 14:49:46 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 78E61140AF8 for ; Tue, 7 Feb 2023 19:49:46 +0000 (UTC) X-FDA: 80441535972.13.84FB519 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf15.hostedemail.com (Postfix) with ESMTP id 6391EA0006 for ; Tue, 7 Feb 2023 19:49:44 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=haYUqGJq; dmarc=none; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675799384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cNa9Sf0hm/zto1MhMuM31kpEGjxz53JlMber8bX7Qfg=; b=1jwlDiP+HcS5RNSapOpNK4pMrst9bXTVbCM7hEFerIzv/gjwCPIp1qcniMAr4R+E1vPydt VwUCJYA7pvdZukaAUcu1y9wePijHU5IsCwXIoXmr5NM/95GINDi83iQQ+pgBuudXkRqggR Xcww+odwaCXPdiWGyUhcbxiaOCtfu7A= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=haYUqGJq; dmarc=none; spf=none (imf15.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675799384; a=rsa-sha256; cv=none; b=fBmGRtPOxgf+FfiMkt158i5yPGskIuxEnkbxPbMhjgw9f2ZBFQYDDa76SAAQb7QULgLDGx AEieSLDYyQ6n3r0w+XJc5k4tls3dkptxqfK3iihAsjTXjBYHFEzPFkKCf9n4sP40OgoclE RSu/O6jtUWwZID3+LrIcoqnqSIzzsFs= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=cNa9Sf0hm/zto1MhMuM31kpEGjxz53JlMber8bX7Qfg=; b=haYUqGJqnjtzUqdedYRaXV/A3j v6WtJGeFjGxzQJag+JnHv1gn2kRvMd0YwwBaXlLSL7rcH+laJGFWncbDqW4qUc6s7b/ROMrQbxduO yaVZlCJpU1d3vwxMytE0/ij+yi8udEBuJkVuCk/M4GOF57ipnsgR6Jce+6wDELbNbro083emDxzEp Z79l1Ovv3DSmCgxiCkxzD3/us7OlVuMvZeHOgYffZeWCQ5+orxuu14nXoRqjAtXWTWyCwYY8e5Jbl h26fiFSAFVnlNt0E0k7k34gT6YTCOx3aVmvyp1cz+w7a1CeZ1Ha2N1CUCsNw6EEsMXHbArXEVqnpz ltHN7mJQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pPTyF-000Vsr-UL; Tue, 07 Feb 2023 19:49:40 +0000 From: "Matthew Wilcox (Oracle)" To: Yin Fengwei , linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , david@redhat.com, dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com Subject: [PATCH v5 2/5] mm: Add generic set_ptes() Date: Tue, 7 Feb 2023 19:49:32 +0000 Message-Id: <20230207194937.122543-3-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230207194937.122543-1-willy@infradead.org> References: <20230207194937.122543-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6391EA0006 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: adaeynsih4phswrew94484d9w1sf393e X-HE-Tag: 1675799384-249297 X-HE-Meta: U2FsdGVkX1+KArkeTHRlP+DLC9aka5YHbXZbvS7ociLhQ0hjRGWV9fgirW6rkaP6nkfXFCIp3AnmFF6wWV3hrPRyTKxUzOBgqqcpIo5mFNtSLNspDL3zCikj+Dolkwq/okPpYsuwOLJ/3dMU1cwomcRRdyC8Lquvh9DIkjbm9zY9M/51DM59RmS2PK+FOJ1YvG2pw8ZJ+pWlNDJGacho5xEBOEO6wlziDz/rmrkfN6pO4B+LmEDSKUhjqc3JxtQrwzHYlyvNHB4UxMJnCuqTcXLNcYkbj8R2891xB3eh1bInW5Cx7ETQSs7hQZf78GrEZ1uSBUHEU7MyscTlt4t3SknmOPPwqikacbjD0c5NUPF8Omw2+HdFHdFi9vQEfMvTU3GNXUstKi1285aogUSfA42qyUcYnflrv8KkelRPljTwZy0ZbuU4QKzWJZORc3MTPJR4vMgobWmpEp7KJu87qJSYBrR7it9xcOXyxjkxEjD+VbFWd1X6UbMBl6fN67mgDjmY1NThmXtYXhsHXy8Vg9YtpMFrmiNDZiF5ytPhD3dWa3nOv/bYtJUndMZAB5cWcvsvFUSvrDqZ/E7p233CEdwYSY8Vp80Nn/gC+a6wmTIr7n2mFKjDm4+6Vdkgk0TwRBY1ijgSHYhzO4+hQtwWcoY5RwE7dpDDJU53NZA4J6CQHI8Slt1Lmsi2QM5Eq3YZojq/IAbqerAvBxT7y/20fEx554N4QgdOqXg3o0Cpt7kSLi3EHFeiHdbsI+gOp17opMe92p5BJUg9FAayHurGTtfotlYsxit4Hyz7DN7OKwYGt1s+RyGy+iq7aDK86UH7L54RbJJcSsTZyk037OUWB92hIWZ817z+bWH13JBBU6rVLW/+GXpUWid7+lt0XBjEFTdlEsa8JuPhP4nVG7uxIIq9u4nhr4FgZPHhK8vMw9C7sY4wkCohL+PMBGPKA/NkSvjvdaXWAu8LFZa0laW QMAVB7Qr sZCTqxWtHkBu7S/cIIpK42cub5sJSoKI5jjoVWO0zDbtzQyqqXQ53qi+hU00zCEHfljgrd/kjBfDhwk4nOYcjySbpiWSNnqKAwzJ9DuJmm1Vx2oebiGR1id3x9REqJNseDqSvOkDHV+QKGVJZzcrwOIGuJDsA2AlPOa72Fhx/Y8SxT8W6DzVbOYVBlOyJMZf5vTxruowmyVkJhzgR/bgER7V0wNovsCpp8ww/aqRVK1QmQH3Y73GiY+dcFw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This set_ptes should work for most architectures. Those that need to set a special bit in their PTEs or have their PFNs located at a different offset from PAGE_SHIFT will need to override it. Signed-off-by: Matthew Wilcox (Oracle) Tested-by: Yin Fengwei --- include/linux/pgtable.h | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index c63cd44777ec..e1804d23e7c4 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1439,6 +1439,33 @@ static inline int pmd_protnone(pmd_t pmd) #endif /* CONFIG_MMU */ +#ifndef set_ptes +/** + * set_ptes - Map consecutive pages to a contiguous range of addresses. + * @mm: Address space to map the pages into. + * @addr: Address to map the first page at. + * @ptep: Page table pointer for the first entry. + * @pte: Page table entry for the first page. + * @nr: Number of pages to map. + * + * Context: The caller holds the page table lock. The PTEs all lie + * within a single PMD (and VMA, and folio). + */ +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + for (;;) { + set_pte_at(mm, addr, ptep, pte); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + /* This works for x86. Check how PTEs are encoded */ + pte = __pte(pte_val(pte) + PAGE_SIZE); + } +} +#endif + #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP #ifndef __PAGETABLE_P4D_FOLDED From patchwork Tue Feb 7 19:49:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13132090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24855C636CC for ; Tue, 7 Feb 2023 20:12:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C86B6B0120; Tue, 7 Feb 2023 15:12:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 579286B0122; Tue, 7 Feb 2023 15:12:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4410C6B0124; Tue, 7 Feb 2023 15:12:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 31F116B0120 for ; Tue, 7 Feb 2023 15:12:30 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id DF83E40BE2 for ; Tue, 7 Feb 2023 20:12:29 +0000 (UTC) X-FDA: 80441593218.16.2DCE95E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf18.hostedemail.com (Postfix) with ESMTP id 51F5C1C0019 for ; Tue, 7 Feb 2023 20:12:27 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Gr7b9GbX; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675800747; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9h4ihtcgFcuT6F8HLGxBLPscqZGMSJCVAH3r21uKKeI=; b=c65suAiz3F4FXdbUryLvE77aEsM7tvaHmcAyAWHEGWyznpxNnUPbNS+xWq1r00yGCSNqpU nGVs27fI862/ho3Tpj9GrShliQ/IbafyY1if+vVxka05BhNvekB6Fg1s0a5cG2TXN6upOe XvjkLoAMe9YWVjx84sQ73eUKP0OgIfQ= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=Gr7b9GbX; spf=none (imf18.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675800747; a=rsa-sha256; cv=none; b=2fsmGFCT5GD4ySgmT4/e374LbjgLP+f7X2pL/iHET/85cLUT93V4DlpO+e8yHEDBK+No+r 1LjpY17ZnutcfGphZOI9rJYRFm7HY/yrom81/3pWdSUkksgUPKmW1SdspBPfdVGWb3g6YT CAcMNRWwOM7agC7FbY39TCB0RAs+bdM= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=9h4ihtcgFcuT6F8HLGxBLPscqZGMSJCVAH3r21uKKeI=; b=Gr7b9GbXb/LRP25MZ3pMW/XjDR nQTZzn5/ka6S3ydQyEjhW5WS4J6rs/D5/7iTtSuHywRA5GeoWErLkLi1cSU4yciyayHzPfBCxFhbE qnMTLvAZGNHwXw/rWMGRm1SMNmPGUKc9p1YEglJ6pneWzVlEJxMC/5nSwuDdvgY9OBEAafGIRXCRc kZJ0Jof+pl08vbRssxUmYV+K2xRL+uarpIJO11OedGddHVXo+CU5Gqp/DX/HjU4ksi0GPoHTahDMX 2DuFGVi03CsMkdmNAS3+MivSNSE7PiTkGyz46QjFSR3WRbQVMv4twpwCHLUklRJ/fmBI/eeaU/C9v r7Cm2RhQ==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pPTyG-000Vst-AE; Tue, 07 Feb 2023 19:49:40 +0000 From: "Matthew Wilcox (Oracle)" To: Yin Fengwei , linux-mm@kvack.org Cc: david@redhat.com, dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, Matthew Wilcox Subject: [PATCH v5 3/5] rmap: add folio_add_file_rmap_range() Date: Tue, 7 Feb 2023 19:49:33 +0000 Message-Id: <20230207194937.122543-4-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230207194937.122543-1-willy@infradead.org> References: <20230207194937.122543-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 51F5C1C0019 X-Stat-Signature: i44c818dcfwtgdrg48a5idj93ueqpnkh X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1675800747-552078 X-HE-Meta: U2FsdGVkX1+aOC+BotPt4AzWGndjfsDWttzmfGDD5hA1NQf4cYgfBNmaC/FTtswmMiU8h02WR41UUHHWMon1tdyf94sXSvJ83RniIH8WO5cH5AZPHuiVa1SpcuqQYEjISodeOkAm5hz1YQrMsHEg88iEI1yGY8BndIYaeLs8ihFM2anNCudDZQGL/VFmfUgxpCwuXuKVhewDJ8MrQ+T6YfKQ5ZeggLzvReewkMB1iDLCSFXKr/DfZIYhI06lnSDfvf+N769gipp2Ohp8ZZleD4KfV1UuwC7Lk6LSFF9U/IYLLIqR9CzheQTKK+M/aBlfbIv7qiZ5e3l3Img6qEW5M9/HOWLNMsJUXd24HVJteAACinGwIRp0Njb0GR799hPYvVmUYpmJtXPT6C+OVe9Bn8rUO1q/NThwmo7+jMrXgjSY5zgNyVUtax6qxaOSa+XCYBv96ZaKicjmia2uprxTz13znKR3zhMPAQfMy3CCQnSNedDLio1r2/TFxhT6yAFRvh71z10sM6uwbrejDtCft/FZTLXNG+MNTUmrCa4mWsjNJj/sPJ5+iAUWaULPcWK7PI02fSrfgziLi7vHQN45DohVsn5LsT0BhNHvJlTUX28PUnrtDecpPpwuMF74Qpwzl0h0ukfKnBsIHsiSq8MEOF8zHTIi+0kcf1EDWzPN7EnGYEBE4ynbjsBLPepIn8ofOZ8yoL+Z31f4XTE+TAwjHAqIyRV0OvqQKi0LROmiUWy8ZqM4xTBJNh5pyL8oo58FlWdbDPvwLY2JpR6lXIpo91Y+d65uRrzhkb7BC9bysswyH7OIr47jErqCx2iswO5qhcpKJCxX3+X3ApQl9+vHATJWf9XXnzPvigxWtjs+HOo+GTl7qbtwcj1sbzeJZfLLmh40E7nulO6/STAZTYqhRwD8aoZ1rCcLlNoDeBVh1QBLz8Y98RT36+ikaAC4TULFmsz9hh2ZBirX7VOZ/Ih Mj41brxC Wep1O86TUqIRCHwWr4XcZg4ACyhrccHRm5sFTaAyifIJLivJ2XKK0ueuUTrgWYELqWRd64nMHsxA/Dkd0U6JWlB7bRlYXH9xH8/n+o1zYbnRW1uELhWASs2c0D6pMPRXm09pOBizG8PgBG5q3burbVhTOZ2odbkg4WxsEIISRrWCjOexSlkD0Dm2yAQmPwTSSmRsBLCUV5yVLjTZpLQZB80o8AnfWkIEwUtXZtGbY0RYVzIxzAydh/XLx/DPalHG6oDv0u4jV4Kw6mq8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yin Fengwei folio_add_file_rmap_range() allows to add pte mapping to a specific range of file folio. Comparing to page_add_file_rmap(), it batched updates __lruvec_stat for large folio. Signed-off-by: Yin Fengwei Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 2 ++ mm/rmap.c | 60 +++++++++++++++++++++++++++++++++----------- 2 files changed, 48 insertions(+), 14 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index a4570da03e58..125ede2fb049 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -198,6 +198,8 @@ void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, + struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); diff --git a/mm/rmap.c b/mm/rmap.c index d52d90af0a7e..b1929adc4924 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1303,31 +1303,39 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, } /** - * page_add_file_rmap - add pte mapping to a file page - * @page: the page to add the mapping to + * folio_add_file_rmap_range - add pte mapping to page range of a folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @nr_pages: The number of pages which will be mapped * @vma: the vm area in which the mapping is added * @compound: charge the page as compound or small page * + * The page range of folio is defined by [first_page, first_page + nr_pages) + * * The caller needs to hold the pte lock. */ -void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) +void folio_add_file_rmap_range(struct folio *folio, struct page *page, + unsigned int nr_pages, struct vm_area_struct *vma, + bool compound) { - struct folio *folio = page_folio(page); atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool first; + unsigned int nr_pmdmapped = 0, first; + int nr = 0; - VM_BUG_ON_PAGE(compound && !PageTransHuge(page), page); + VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); /* Is page being mapped by PTE? Is this its first map to be added? */ if (likely(!compound)) { - first = atomic_inc_and_test(&page->_mapcount); - nr = first; - if (first && folio_test_large(folio)) { - nr = atomic_inc_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } + do { + first = atomic_inc_and_test(&page->_mapcount); + if (first && folio_test_large(folio)) { + first = atomic_inc_return_relaxed(mapped); + first = (nr < COMPOUND_MAPPED); + } + + if (first) + nr++; + } while (page++, --nr_pages > 0); } else if (folio_test_pmd_mappable(folio)) { /* That test is redundant: it's for safety or to optimize out */ @@ -1356,6 +1364,30 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, mlock_vma_folio(folio, vma, compound); } +/** + * page_add_file_rmap - add pte mapping to a file page + * @page: the page to add the mapping to + * @vma: the vm area in which the mapping is added + * @compound: charge the page as compound or small page + * + * The caller needs to hold the pte lock. + */ +void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, + bool compound) +{ + struct folio *folio = page_folio(page); + unsigned int nr_pages; + + VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); + + if (likely(!compound)) + nr_pages = 1; + else + nr_pages = folio_nr_pages(folio); + + folio_add_file_rmap_range(folio, page, nr_pages, vma, compound); +} + /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from From patchwork Tue Feb 7 19:49:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13132078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55F5EC636D4 for ; Tue, 7 Feb 2023 19:49:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3D516B010F; Tue, 7 Feb 2023 14:49:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DEE916B0110; Tue, 7 Feb 2023 14:49:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C904F6B0112; Tue, 7 Feb 2023 14:49:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B72096B010F for ; Tue, 7 Feb 2023 14:49:50 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8A9C7AB154 for ; Tue, 7 Feb 2023 19:49:50 +0000 (UTC) X-FDA: 80441536140.13.AC1305D Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf28.hostedemail.com (Postfix) with ESMTP id D70F7C0017 for ; Tue, 7 Feb 2023 19:49:48 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=HHpeaWbP; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675799389; a=rsa-sha256; cv=none; b=OgwmJgkipCb4z4ekupQ6Ood7q+J6pFCE8sJbFHfIZPD/UqBqYvjg1+AHG3M79f9sOUqu2h 5tUyGQmqdpLO/pI4tRJWLkcDQwHRG9rnFSxwHYp09N48tB4W1wox2Ws9blnaZtJjS7j+YP bzI9TDYCQ722B7cL+e1A83YIePmufHg= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=HHpeaWbP; spf=none (imf28.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675799389; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F40oIATo9rCV1BRwMZ5Hk4sakjKgHGGGz9UmCl1t1Xo=; b=iEF6/H3s6I/rbOMRBKZxY7H0oU5HZsoKq0ai6iN3VgTTRXoXaEI9iY+9kA4cTrDHZGxPAf peHLxroiu68ZC37m4vM+U0or+pH2W/t9DFrAPbDYnAEtbiDRLp8G8tFwopujwvUz8qYHvU eZpotNOFshCTM6fz4EIAYOWxvmckSxk= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=F40oIATo9rCV1BRwMZ5Hk4sakjKgHGGGz9UmCl1t1Xo=; b=HHpeaWbPuZ6SY90brAJcdeS99b DoX71Snz48HKnqqF3abdRQupNeCW0qGpUWnVxsfyGvruMnlXct/020uSDmxvi31pFJPq/ojCIs7Ao 2E9kArVSMF25ZJLaZhJhxeoit2OxTe4jpKJh8WaoB+6SSdvcziptWAzsNpJNGmYlon+M0onJcatwC 3B3cnuAaXhwE3STP1MdxLqQlRqjAw6t0yp3b4NH9NQMSiDN3exu9ZOwpPB77k0ugYF9VXyYQsY201 kcj6azlv5jj1N+2kHhF1K/0OZ3WkpUt6N145WaH1DFjpCpmx1hy6QSpMZohdB9yliTHYs5JZ+GTCx Rh1/ujzw==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pPTyG-000Vsv-PL; Tue, 07 Feb 2023 19:49:40 +0000 From: "Matthew Wilcox (Oracle)" To: Yin Fengwei , linux-mm@kvack.org Cc: david@redhat.com, dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, Matthew Wilcox Subject: [PATCH v5 4/5] mm: Convert do_set_pte() to set_pte_range() Date: Tue, 7 Feb 2023 19:49:34 +0000 Message-Id: <20230207194937.122543-5-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230207194937.122543-1-willy@infradead.org> References: <20230207194937.122543-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: D70F7C0017 X-Rspamd-Server: rspam01 X-Stat-Signature: nigriwfp8qnx5qzp1i4arkptyyoj59u6 X-HE-Tag: 1675799388-81633 X-HE-Meta: U2FsdGVkX1/JT2MD1Im/7rWurw3tPr9tBlHTrELZ3cogZIY32g1PWKRdGXgerQSMDyxZ2LKpcu2micTm8CQoRHVtdexF/12mogNpGTmwYPognIpY4BZ98Xr694GoJW+iSDDDD/ven+WiSgNTx8QKfo+tEyL5oUfMM07SArPgglnd9Y4v4bmdpf1y2+a4SRDiaIC7RVSlT5ehA5nITWV9ivFyKqDN4Fw7QeHachX16P9PBrm6lbzGGamWQN8GhiDFdi0s3dlHUsCSedonZpiC8R4ecHazLVEmcFm6DbLB4eVdUDJIkFI9chix8SVDxrMROLRbvAz0LRhwH3u3QJY/lNh5M9SAZQTnVK8XQibkxjJhQysGQgEC6r5rM7XqCsHxYkPWNi5jxbPnBG12jhyLbvUBWs5kLrnVxIx1ZgaDGEsY/cX4nK5SXFI0aQLhGK2p6D36h1NcDLKb6hbFWFOvnNlh1Wu4WyS6gXTVfcQF1+HxY7qbqI2Hq8Ir11zNR2g2H5z4CcmqoHtXp23Vty2kbKkY0W0YBwcpz9o6kNmCupUbrttmAtG+RS3xXyRq9eZsvWpHIgdl7FMoRLdc971eCElmrZdDp86OwqT53gq4yoyBH4eXDtuMYhaNqx4bqWZLF48liI/JMbgklqipuOjtB4TEKYjSvfwZU+HOreyEKVqqTrdYXQSXI7pyE+A8JbhcsnGtQ+We4Y/fxSYRJV0WX7tcN/UL9chmvjZQnfHraH64e3+VkizEQpFe/kXuyiGgPgXbS1UYHVSWIM7qFJj9gpNHrf+6UtNGeyJfHg2K+rYm3khSZBddBr7R/lWpGYyuUTOF1PBzbvz53dJEF4y9LHJkfEEibXpVmJCVUe5DkBDijabaZNsYx31R9lDr/7EsrWhahnqQ9enl87tJrURt0KIi3CGkPy6F4sZ6ZZ0/RT1VVOCT/dVCOihC1Os3qN8Q0Qov5JC34qAvbnRj4oC n5ZFkUuX EWV17KeO9geOnyVGbWi3Nujxy5nXWLcHXEG8RDsN/J2MyIWpq5DzoiPQOfCPrDOUuoJ+ZfCI3QIDYRxiecE7+LfiSaupKp3epr9jWeDMm8sbdFV8LY9vy/IG6EGW9wp1snAes1fUwzYh5aZueMwNHpLCtMFj4ESWWRussuyCqOQHAe7RNsb0Hu+4VDPqkQlUmTJV8aQXqqYhOrcIixmdROMpJibtQAJmpMtX5q6UcNl5QKIdeW026yo/e4q2GwT31abBXvO8rqWh7ymc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yin Fengwei set_pte_range() allows to setup page table entries for a specific range. It takes advantage of batched rmap update for large folio. It now takes care of calling update_mmu_cache(). Signed-off-by: Yin Fengwei Signed-off-by: Matthew Wilcox (Oracle) --- Documentation/filesystems/locking.rst | 2 +- include/linux/mm.h | 3 ++- mm/filemap.c | 3 +-- mm/memory.c | 31 ++++++++++++++++----------- 4 files changed, 23 insertions(+), 16 deletions(-) diff --git a/Documentation/filesystems/locking.rst b/Documentation/filesystems/locking.rst index 7de7a7272a5e..922886fefb7f 100644 --- a/Documentation/filesystems/locking.rst +++ b/Documentation/filesystems/locking.rst @@ -663,7 +663,7 @@ locked. The VM will unlock the page. Filesystem should find and map pages associated with offsets from "start_pgoff" till "end_pgoff". ->map_pages() is called with page table locked and must not block. If it's not possible to reach a page without blocking, -filesystem should skip it. Filesystem should use do_set_pte() to setup +filesystem should skip it. Filesystem should use set_pte_range() to setup page table entry. Pointer to entry associated with the page is passed in "pte" field in vm_fault structure. Pointers to entries for other offsets should be calculated relative to "pte". diff --git a/include/linux/mm.h b/include/linux/mm.h index d6f8f41514cc..b39493b5c49e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1161,7 +1161,8 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) } vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page); -void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr); +void set_pte_range(struct vm_fault *vmf, struct folio *folio, + struct page *page, unsigned int nr, unsigned long addr); vm_fault_t finish_fault(struct vm_fault *vmf); vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf); diff --git a/mm/filemap.c b/mm/filemap.c index 9aa7a4fdc374..745f1eb2a87f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3375,8 +3375,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, ret = VM_FAULT_NOPAGE; ref_count++; - do_set_pte(vmf, page, addr); - update_mmu_cache(vma, addr, vmf->pte); + set_pte_range(vmf, folio, page, 1, addr); } while (vmf->pte++, page++, addr += PAGE_SIZE, ++count < nr_pages); /* Restore the vmf->pte */ diff --git a/mm/memory.c b/mm/memory.c index 7a04a1130ec1..f1cf47b7e3bb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4257,15 +4257,18 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) } #endif -void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr) +void set_pte_range(struct vm_fault *vmf, struct folio *folio, + struct page *page, unsigned int nr, unsigned long addr) { struct vm_area_struct *vma = vmf->vma; bool uffd_wp = pte_marker_uffd_wp(vmf->orig_pte); bool write = vmf->flags & FAULT_FLAG_WRITE; bool prefault = vmf->address != addr; pte_t entry; + unsigned int i; - flush_icache_page(vma, page); + for (i = 0; i < nr; i++) + flush_icache_page(vma, page + i); entry = mk_pte(page, vma->vm_page_prot); if (prefault && arch_wants_old_prefaulted_pte()) @@ -4279,14 +4282,20 @@ void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr) entry = pte_mkuffd_wp(entry); /* copy-on-write page */ if (write && !(vma->vm_flags & VM_SHARED)) { - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - page_add_new_anon_rmap(page, vma, addr); - lru_cache_add_inactive_or_unevictable(page, vma); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr); + VM_BUG_ON_FOLIO(nr != 1, folio); + folio_add_new_anon_rmap(folio, vma, addr); + folio_add_lru_vma(folio, vma); } else { - inc_mm_counter(vma->vm_mm, mm_counter_file(page)); - page_add_file_rmap(page, vma, false); + add_mm_counter(vma->vm_mm, mm_counter_file(page), nr); + folio_add_file_rmap_range(folio, page, nr, vma, false); } - set_pte_at(vma->vm_mm, addr, vmf->pte, entry); + set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr); + + /* no need to invalidate: a not-present page won't be cached */ + for (i = 0; i < nr; i++) + update_mmu_cache(vma, addr + i * PAGE_SIZE, vmf->pte + i); + } static bool vmf_pte_changed(struct vm_fault *vmf) @@ -4359,11 +4368,9 @@ vm_fault_t finish_fault(struct vm_fault *vmf) /* Re-check under ptl */ if (likely(!vmf_pte_changed(vmf))) { - do_set_pte(vmf, page, vmf->address); - - /* no need to invalidate: a not-present page won't be cached */ - update_mmu_cache(vma, vmf->address, vmf->pte); + struct folio *folio = page_folio(page); + set_pte_range(vmf, folio, page, 1, vmf->address); ret = 0; } else { update_mmu_tlb(vma, vmf->address, vmf->pte); From patchwork Tue Feb 7 19:49:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13132076 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D6ABC636D3 for ; Tue, 7 Feb 2023 19:49:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA49A6B0111; Tue, 7 Feb 2023 14:49:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D02F26B010F; Tue, 7 Feb 2023 14:49:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B7C7B900002; Tue, 7 Feb 2023 14:49:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A64946B010E for ; Tue, 7 Feb 2023 14:49:46 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7BB6BAB17A for ; Tue, 7 Feb 2023 19:49:46 +0000 (UTC) X-FDA: 80441535972.12.5C0D0D3 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf01.hostedemail.com (Postfix) with ESMTP id 6761F40008 for ; Tue, 7 Feb 2023 19:49:44 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jiDG9UC6; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675799384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RwrLOQlQFZldBSoSj0NqBnJnjxxeVCBmAtcieBK8nys=; b=PydsKObeRIhuJmfb9p03M4pH3E26SUkAe6w2On6ERjso1iVVOwWPdQAwU9XggWlctulAzr WlxX5RRVeJa2eLVpwnfxZVqA9gRiK1mLliUEhUonF87IaIISP1fE9uC9DVuXysyt00mQNd kpDZdX1NkGS/mQwaCndgxZqtaHOpKDs= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=jiDG9UC6; spf=none (imf01.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675799384; a=rsa-sha256; cv=none; b=VMreUm+RbDpIK/C2oFjEQCyFWQKPshV1msQBGfiQeLks7m7moy5G6niERdaHxoY1zvoFhu ++73yp6a+7s1X61ckIOpERuHCMApv6pf46lvh5tRyy/XUiZPILSWl0K1Omilh9gYdZ0hQN gfeN5zTDysujYh3KsnM7SYRp3xgxiPY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=RwrLOQlQFZldBSoSj0NqBnJnjxxeVCBmAtcieBK8nys=; b=jiDG9UC64itoB7qW+fvXO/P9Wi s6hd3Sq9wsR1k4OFUeEwoRq/en5nDc7lkB2dz9F8wUtHky8pEJy2PgeEvNU1lu5DzNVwMnXsmmwhg hLiPo8FksL+3Icm1Q+nz1Fdn/i4zIQmuQ7hh5BoFwdRzxAA+AOh5c5SGEBK/k93VXYBqgRuTttVSD hXCLD7C+M2tg9Kz4lN0CL4Gi6JC6WbOqzW7AVHRlpvvGPLnIlUQuOBFcBlLyOV/93qtdWtbDWAMK7 a0UgtF/tFtdcqcmmIVP/6IQh2D3jwUC+a2zTklVhyb18d3qgIezijrnqWVv8YsPKH/Bf+THlXtEFk ECsHt78w==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1pPTyH-000Vt1-8N; Tue, 07 Feb 2023 19:49:41 +0000 From: "Matthew Wilcox (Oracle)" To: Yin Fengwei , linux-mm@kvack.org Cc: david@redhat.com, dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, Matthew Wilcox Subject: [PATCH v5 5/5] filemap: Batch PTE mappings Date: Tue, 7 Feb 2023 19:49:35 +0000 Message-Id: <20230207194937.122543-6-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230207194937.122543-1-willy@infradead.org> References: <20230207194937.122543-1-willy@infradead.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6761F40008 X-Stat-Signature: caibjnsbo9iy4xsxo13tn18grygajwwc X-HE-Tag: 1675799384-230873 X-HE-Meta: U2FsdGVkX1/VNGjwl1fvnY8eH0jqQokGYsZ0UXEOlkHUR3Ls1VrVxjpbW8/jH0tBS0+m9QVypzkbyITD8m3pQSww15MOM+tfxIm1mCJlSq4GG8IzVKKcfqFH8Q01ho8WwBlwDgwhZOc34ldsLpvSReO4BlY4konhrDHF0o5gFmjtGaygsHgC2cxOQa1twgbDMiDZ3atuwNoCnGUB4GNnAAWLbe89pv4LQHm7hKIUPOcXxUtKhLU3pZLepQunPKXW28OpIEPI9QSEHoRceW2DhsB0Ve879JdggNp5wOdKC8OYBRGfQXTZ5lpsUrHbd0o+wfnf4zGR3EibjZ8einbitZcKK34jpbTGwizVKhGf19AcBEGdRNMPVBmokFYP6FpL+EIz6KgpVrWpnI++J2a6vUW+bhUxy8Fg8lyBGFhpPkteIvFtQad3B2z6+4ZuzGk+HKzHzYb8aOWVUenDZPG6DJ4LJHTQfCxgO6m3LHcGQH5A7uYUttg5bq/kuF22MShHcg56IIc1BaWOgoPgE8ABPaq7OW7KfdMXNO8vVKXWmVd281Ij8q7mN6KjQ8HGQ/be2ulKHRzrNtIUJXzo8GBlZRrgZC02n6Oerk+1iutNUEjwYrnW0nJggmhn4oGoPsExeqAAf4jZGwzgQ50PiVH2fkn56iYDee8Bm4RmidV/f5r5D74XJ84zAk8mGnVd7n7RBJjwNKbSJWZ9h76mzQRRGB4sDRUl3QC62pW2lH3qKRtVVfCv/q1HfHzyTE1UHE3ANW4ffVlDL0TmIkPh+BO0zOxZzs1ms5cteIxTSDBiLuzb63HLcbTWWfbBcM7V9nIL3oVd6aSECZAZXG5efE0fzlbdefa/PEuMowt+BKRulYwNfTG/GB7mXN0Cu2l094BQnIoj+okoa7ADxsko9OPxu49CXkPB6WuY/xHB24HpqdPV9TEk4uaGOS6UY/yy4MQuM/zkQ0blUjms6rORF8/ lUZmaOzG 4FBRZ9Feqpy85T/6A4Tb8scS147svMwaeozjMjWC5h6EBviFBuAZm+3BXzmsxr1ZMs4oxKbSSho53uIK31QCradDOLJKri7foMcK9KXDGFB0AjoHrOP+hYy07L7uU6rX1UdZ46Q+gxAH6JLH6h6etfnbPYKrjThbYyUrAP1f1kXt4mMtt7pkwVgIHiOrapaEXnzIqo//RlcNZY3DDdsDqYckHRY9PXg+P+xDcQ9W0PAnMjAfflSY0+LUDug== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yin Fengwei Call set_pte_range() once per contiguous range of the folio instead of once per page. This batches the updates to mm counters and the rmap. With a will-it-scale.page_fault3 like app (change file write fault testing to read fault testing. Trying to upstream it to will-it-scale at [1]) got 15% performance gain on a 48C/96T Cascade Lake test box with 96 processes running against xfs. Perf data collected before/after the change: 18.73%--page_add_file_rmap | --11.60%--__mod_lruvec_page_state | |--7.40%--__mod_memcg_lruvec_state | | | --5.58%--cgroup_rstat_updated | --2.53%--__mod_lruvec_state | --1.48%--__mod_node_page_state 9.93%--page_add_file_rmap_range | --2.67%--__mod_lruvec_page_state | |--1.95%--__mod_memcg_lruvec_state | | | --1.57%--cgroup_rstat_updated | --0.61%--__mod_lruvec_state | --0.54%--__mod_node_page_state The running time of __mode_lruvec_page_state() is reduced about 9%. [1]: https://github.com/antonblanchard/will-it-scale/pull/37 Signed-off-by: Yin Fengwei Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 36 +++++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 745f1eb2a87f..4abab43d9564 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3354,11 +3354,12 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, struct file *file = vma->vm_file; struct page *page = folio_page(folio, start); unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); - unsigned int ref_count = 0, count = 0; + unsigned int count = 0; + pte_t *old_ptep = vmf->pte; do { - if (PageHWPoison(page)) - continue; + if (PageHWPoison(page + count)) + goto skip; if (mmap_miss > 0) mmap_miss--; @@ -3368,20 +3369,33 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, * handled in the specific fault path, and it'll prohibit the * fault-around logic. */ - if (!pte_none(*vmf->pte)) - continue; + if (!pte_none(vmf->pte[count])) + goto skip; if (vmf->address == addr) ret = VM_FAULT_NOPAGE; - ref_count++; - set_pte_range(vmf, folio, page, 1, addr); - } while (vmf->pte++, page++, addr += PAGE_SIZE, ++count < nr_pages); + count++; + continue; +skip: + if (count) { + set_pte_range(vmf, folio, page, count, addr); + folio_ref_add(folio, count); + } - /* Restore the vmf->pte */ - vmf->pte -= nr_pages; + count++; + page += count; + vmf->pte += count; + addr += count * PAGE_SIZE; + count = 0; + } while (--nr_pages > 0); + + if (count) { + set_pte_range(vmf, folio, page, count, addr); + folio_ref_add(folio, count); + } - folio_ref_add(folio, ref_count); + vmf->pte = old_ptep; WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss); return ret;