From patchwork Mon Feb 6 14:06:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13129995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B28CC05027 for ; Mon, 6 Feb 2023 14:08:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C74B36B0073; Mon, 6 Feb 2023 09:08:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C23A36B0074; Mon, 6 Feb 2023 09:08:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D9256B0075; Mon, 6 Feb 2023 09:08:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8D2056B0073 for ; Mon, 6 Feb 2023 09:08:37 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 418C5A7EC0 for ; Mon, 6 Feb 2023 14:08:37 +0000 (UTC) X-FDA: 80437047474.01.C29B575 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf22.hostedemail.com (Postfix) with ESMTP id 1474DC0026 for ; Mon, 6 Feb 2023 14:08:33 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=K3NSK7Wc; spf=pass (imf22.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675692514; a=rsa-sha256; cv=none; b=cyGT1UIpviIV+6WlKajRgy+VxcAhaSqCpNU60FGiSPhEjwx1MrmLVoVlgbvVRhG4IMSKrq Ykn8CkwiHVwGtJzOnT8n6MzqbqY0VJIjcR0NSVP0bB84c8fRUXQvX8G2LObtCgfXPZAV7N wJqYFBLEWG6mEltLtn+Bcq7IlOaqG5w= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=K3NSK7Wc; spf=pass (imf22.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675692514; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/uVWC5FLEjWB0RQqWV9EAE1R4d2ffmf+UvJEBOHdal0=; b=KzQZMUdIz2c0OGVQyQvLcGy9KSCS+UHVMA1GhAfnp/udd+UJCikkD0OPY552SaesHJWB5B Rhjw5rgsRWddB4s/otS6YcmMMLIxYnhJn0vTrMZ5YXgMOpuGTmzjgH0//QLkUcsqYD2TA4 1qfZN9sjOI8HL5Q0F8ZU79ox5Oa+XY4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675692514; x=1707228514; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=O1FTSdHZHAOvBAMWOKiLjSHXcjkYjftKN6JqXEhmhuY=; b=K3NSK7WcjgCaTTgcWaRRnlJX5ScXl2IVgRBLXsjpMhu9rUg/QLtZVoq1 tPUSK/yYI51BREXGkPnlLW+8s9v9T+DKvDu6de6B8Mt6yPPExztyi239q M2DBvsQ4pW3Be0vFn5jZ+HYnalh7FkqvRmjwdgkctotUwKadpP3coyacZ xwiCbbgzJj53GXiIFXSI/TA/xUlTfLAC0btmU9D0QnB2+Or5Ds4Uxww4L UMHI4ZoB0NsS/JDiM4+nQqbdHAI65sjCcbi2i3HLZi1IejWsv00GVwcs6 /XNuhB1autsJ9I0UbYu5v5iHbuVlbMtiki7ju8nbuSwky88/T7qPcKEU5 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="312864201" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="312864201" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Feb 2023 06:04:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="911937868" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="911937868" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga006.fm.intel.com with ESMTP; 06 Feb 2023 06:04:45 -0800 From: Yin Fengwei To: willy@infradead.org, david@redhat.com, linux-mm@kvack.org Cc: dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, fengwei.yin@intel.com Subject: [RFC PATCH v4 1/4] filemap: add function filemap_map_folio_range() Date: Mon, 6 Feb 2023 22:06:36 +0800 Message-Id: <20230206140639.538867-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230206140639.538867-1-fengwei.yin@intel.com> References: <20230206140639.538867-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 1474DC0026 X-Rspamd-Server: rspam01 X-Stat-Signature: oe3qqgy73cncxxtusgi153q3cad5tusq X-HE-Tag: 1675692513-125103 X-HE-Meta: U2FsdGVkX1/1A7ZZUehieaY73TnWVtWOQhzyfha5CtCLGFQXLLOLCgpgsqiAH01fPubbQp7/onj5zTCk+lmH2hLtt3Ud3xwHAqMl5sAwqQ+lhhMsZ6B+nWEX9Y6DGmHl3LrG1kJIVkgo1Xvwgu2XWjxDxEwHgHlfqH09QEXYBaaevIrUVmTfcXan8dyu50w62ztObxIApcwuDj41XSd/V1FebgfY3mhXtoz5QGNOfA6IpNsFC0OMYQ1yTPNa3RFDnE8mIH7sDPair5aiXlE0q0M5VlhDF3JVFAV635aOI7IhbOp0mw9Jke1P9v7e5kT9YZXshJ9NlzWxyLoNFhBEf9SrPaAa82GhQIdWF4g+W6JakG0J7Ine1EN8RUfsZf+l/X72vtW9cN9JP0QPHHdNuWravmGNrDpje4X5pkQM67BBINxnUSgRoC39MQh+AJ+MaP/QzMGXnvjTBZz3g60dZfjD8CLKivjc4aFTvIWd0luRLh6xkmWn21EyYb0QZHUupUP/q46MDlciWSMmArzNZ7ZtWm75Kd6SRds36DjGLvh5v4kA7BDHvUpMoPEcxuf5Pu90MSvzW1fmkIeMJwhSZ1EXRcnZpABWt8Z9XLsRNeWt2sOWmhcT43FyKdSy9O2fW2nEFFia2Uv9XdCb5pdzJ9TltJUM/7kj5+zDmBvYuirv32V5trNixBip6ypjReBWAmccCCUv7EMuFqQgMi4iFvpOqLiblFghmJh0oY6a2/5HiqCePKLm4gfDVlPkqAylexPXPo2vUUq/KkVDvT2qQQJb+eO+orYQOlk+lyiHVvS85st2LjD4TrnDpldY/TyPmm0ETtl0/1/BgKwYM0bUqS3j8uQPP92DiBUs+msaxsj9hBqK4Z3aXeMXrdFnOaRyGFJB1nBOVo2hU/lepCi8w6vC21KrGyc5ks4dkKgJj1TUH6Wms6HR0mFswS2d+nOICVnROMSkSzp65iWIpSM hweY4RIG W1Id/4IQsirMZaswoD3YoGoyhc2clHx32RvmubeC+VR8tFdt1GtmS4eeKGer/Xv6ACYaRkqye3A6TMFJMMV1OPnELMptdQwbow0MvohoHRhdpaW2h2UVfK36nM/9c6gmDM44KYEQuA+r+Q04+eiciIb8x5bshYdSyF6Igj4LYtSaGR3n6M2G9mTlwNXlIk4cghdjVxG1SN8sputWt7q8NoFubNfi8jWsJrib8mVTNS9AntOT8yRp7rgttKQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: filemap_map_folio_range() maps partial/full folio. Comparing to original filemap_map_pages(), it batched updates refcount and get minor performance improvement for large folio. With a will-it-scale.page_fault3 like app (change file write fault testing to read fault testing. Trying to upstream it to will-it-scale at [1]), got 2% performance gain on a 48C/96T Cascade Lake test box with 96 processes running against xfs. [1]: https://github.com/antonblanchard/will-it-scale/pull/37 Signed-off-by: Yin Fengwei --- mm/filemap.c | 98 +++++++++++++++++++++++++++++----------------------- 1 file changed, 54 insertions(+), 44 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 992554c18f1f..1c37376fc8d5 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2200,16 +2200,6 @@ unsigned filemap_get_folios(struct address_space *mapping, pgoff_t *start, } EXPORT_SYMBOL(filemap_get_folios); -static inline -bool folio_more_pages(struct folio *folio, pgoff_t index, pgoff_t max) -{ - if (!folio_test_large(folio) || folio_test_hugetlb(folio)) - return false; - if (index >= max) - return false; - return index < folio->index + folio_nr_pages(folio) - 1; -} - /** * filemap_get_folios_contig - Get a batch of contiguous folios * @mapping: The address_space to search @@ -3351,6 +3341,53 @@ static inline struct folio *next_map_page(struct address_space *mapping, mapping, xas, end_pgoff); } +/* + * Map page range [start_page, start_page + nr_pages) of folio. + * start_page is gotten from start by folio_page(folio, start) + */ +static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, + struct folio *folio, unsigned long start, + unsigned long addr, unsigned int nr_pages) +{ + vm_fault_t ret = 0; + struct vm_area_struct *vma = vmf->vma; + struct file *file = vma->vm_file; + struct page *page = folio_page(folio, start); + unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); + unsigned int ref_count = 0, count = 0; + + do { + if (PageHWPoison(page)) + continue; + + if (mmap_miss > 0) + mmap_miss--; + + /* + * NOTE: If there're PTE markers, we'll leave them to be + * handled in the specific fault path, and it'll prohibit the + * fault-around logic. + */ + if (!pte_none(*vmf->pte)) + continue; + + if (vmf->address == addr) + ret = VM_FAULT_NOPAGE; + + ref_count++; + do_set_pte(vmf, page, addr); + update_mmu_cache(vma, addr, vmf->pte); + } while (vmf->pte++, page++, addr += PAGE_SIZE, ++count < nr_pages); + + /* Restore the vmf->pte */ + vmf->pte -= nr_pages; + + folio_ref_add(folio, ref_count); + WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss); + + return ret; +} + vm_fault_t filemap_map_pages(struct vm_fault *vmf, pgoff_t start_pgoff, pgoff_t end_pgoff) { @@ -3361,9 +3398,9 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, unsigned long addr; XA_STATE(xas, &mapping->i_pages, start_pgoff); struct folio *folio; - struct page *page; unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); vm_fault_t ret = 0; + int nr_pages = 0; rcu_read_lock(); folio = first_map_page(mapping, &xas, end_pgoff); @@ -3378,45 +3415,18 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); do { -again: - page = folio_file_page(folio, xas.xa_index); - if (PageHWPoison(page)) - goto unlock; - - if (mmap_miss > 0) - mmap_miss--; + unsigned long end; addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT; vmf->pte += xas.xa_index - last_pgoff; last_pgoff = xas.xa_index; + end = folio->index + folio_nr_pages(folio) - 1; + nr_pages = min(end, end_pgoff) - xas.xa_index + 1; - /* - * NOTE: If there're PTE markers, we'll leave them to be - * handled in the specific fault path, and it'll prohibit the - * fault-around logic. - */ - if (!pte_none(*vmf->pte)) - goto unlock; - - /* We're about to handle the fault */ - if (vmf->address == addr) - ret = VM_FAULT_NOPAGE; + ret |= filemap_map_folio_range(vmf, folio, + xas.xa_index - folio->index, addr, nr_pages); + xas.xa_index += nr_pages; - do_set_pte(vmf, page, addr); - /* no need to invalidate: a not-present page won't be cached */ - update_mmu_cache(vma, addr, vmf->pte); - if (folio_more_pages(folio, xas.xa_index, end_pgoff)) { - xas.xa_index++; - folio_ref_inc(folio); - goto again; - } - folio_unlock(folio); - continue; -unlock: - if (folio_more_pages(folio, xas.xa_index, end_pgoff)) { - xas.xa_index++; - goto again; - } folio_unlock(folio); folio_put(folio); } while ((folio = next_map_page(mapping, &xas, end_pgoff)) != NULL); From patchwork Mon Feb 6 14:06:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13129996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FF2BC636D6 for ; Mon, 6 Feb 2023 14:08:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4D5276B0074; Mon, 6 Feb 2023 09:08:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 483616B0075; Mon, 6 Feb 2023 09:08:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FD366B0078; Mon, 6 Feb 2023 09:08:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2075B6B0074 for ; Mon, 6 Feb 2023 09:08:39 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DD1501C5EA9 for ; Mon, 6 Feb 2023 14:08:38 +0000 (UTC) X-FDA: 80437047516.03.DD59B3A Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf30.hostedemail.com (Postfix) with ESMTP id 7533780002 for ; Mon, 6 Feb 2023 14:08:36 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jEaizIQV; spf=pass (imf30.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675692516; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bSe1dsEI/cRuQnhHM9hcrT2HZtexocFON22OgJp+EvI=; b=nRV8YMNG5Df8DaItSZUOuesve4Nu+rpE5JL0uPom5TQ2sNMtpzL99r46Fud8l6wFlk12pD 5M9jFhrJwSYuc5z9Tw8Kjh5kkCCERX2FHtaw2Xn3j/AlOuua2MLzJFfLDpvnf9B3M7gXo6 5VxsNhf8ADisTcDWLKYZ8P6auqokCwM= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jEaizIQV; spf=pass (imf30.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675692516; a=rsa-sha256; cv=none; b=mGElC2alDIPTgxsnzyPb2WBkZ0786XWIYZW/9QPBlydAxSYzypxyp26R6mem6l02gelXIi lTsyxBQflSrpJELWWLKZE10xZCVT0ab4BGoKw/1BSbCQRYFvU3OWjb17bBT8yQ2yTsEzi0 gBBnjZmD5hVgQ0h1HGhNSMKuOTWA3WI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675692516; x=1707228516; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RGeWKyZa2H852sBbJ7Gc/HeCa7JW/tXb4rXyzwaxNk8=; b=jEaizIQVRak6INr1Roanc4HnTrkRb9o0JeiDKBM6UATVmAxnk8WtSo+z /BXuOFL95W6ZfuA+VCsNe4NhvoHcS57pfotW6GswwGnSVgyJ51NRb61hV 5lz0xEZsPH4g2V+j1WllPg9CMi1eYTZ0vaeaCFzgkZTJJIHwppHREDqH9 DHKNhKcOeuXjA8Xnmbbgrx1OSjl8lStoaKPiORL7uoopicPZlCPOFo0NB upAhaup9QQkhmxt5zwVFsjYuugvFcytYsl8xMCfpoAsZYy71FVXcX+T0+ Gda7B9KwEoVSs7l66xBMsb6xTUepxxWtEYKyM5gKtETNcidiCv9LcmAUo A==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="312864275" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="312864275" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Feb 2023 06:04:53 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="911937871" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="911937871" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga006.fm.intel.com with ESMTP; 06 Feb 2023 06:04:47 -0800 From: Yin Fengwei To: willy@infradead.org, david@redhat.com, linux-mm@kvack.org Cc: dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, fengwei.yin@intel.com Subject: [RFC PATCH v4 2/4] rmap: add folio_add_file_rmap_range() Date: Mon, 6 Feb 2023 22:06:37 +0800 Message-Id: <20230206140639.538867-3-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230206140639.538867-1-fengwei.yin@intel.com> References: <20230206140639.538867-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: hjy6zzsttx1bhdcwap1zxdt9dbouapcn X-Rspamd-Queue-Id: 7533780002 X-HE-Tag: 1675692516-99810 X-HE-Meta: U2FsdGVkX19ntxj2PU1bmv888vJ8DFAD7ODLtRebQ5cZY15bxyzzik2PO0zTeaznvs8CYD7fGtI+7foOZj9tAr6tllSR1i36eAx8dLTVgPTFY4wlJfFlbGTD9UIZeGzaJDXYLDRZVgF64RzgI7eR9l5CGrweKmfGcf75IqNuudiB7FNb+EwWTy3DTj8lpuz/3wYjjMJMVdk7RET1wDdymF8w/mXiEGkw6iwiqsgXKmsArtcOJWF3hmy8hBbJv5O8qSriVEzvBRuaMUMcBU/ghA1a9Dur5nPsUfo19fUoGxngYgcOpcQ9K6FrBMIcu+qfsXlWFu65O6OScVk5zOH4c9eaDXqSpOZpxnZVogReU7WfSx1+68Xqf/3FIQnwRVN2ispYn3xhBEED37FU069q0eeMFZwVZ3VVERucl10YTT/z2oGprbDU37jzscwnnkY5tGJ5IQQEIcs0TsH2FMZorO0I8nrqVCXQiXzLghmSzUmqSveu9GsqvoO99N15dID/+N/wE88PpUmNvvui7ygoRKgXUptl9eiaBtrcJwSsAJ9F9PbUyJuvOLl3d5MTsGwjoieZ2rPWtz42kvvtddMWrnLEibuxKlAHsfq5hQR6q5Vf4BvFfyBp1QDGoK3TB4/qxNkGWn32YjPdu1k+VrN9RXsf04qYG92rmxabajMA6uUm+dsTND5beWROetcv0Xg4p7JMX57Ui+IU4HHk6phdFMhUv8Fbmu8K99ljtejY1vjaB6A7dhKfWDnwvw+kRRrS+SjqILlQCaDDI12NCxWxuS+uWC7AQZzgvgtf7i1zr924gEnZQWcFydaxD4yfno/BAIkczXT0dtUKfT4RsjaL2KQzGFv5FVqD/ESwf4NcHUUTL28ptJ1Z9pdXKpWYUKYGrdaw/aRqsRUPw/8vN593G84YMFf/OuwmJA6JhVToYsRbd4w/y8MNMNHsFQi4gJlEJSxjWWP/N1Ab02LD6fF n3tJCYCV 5en8HHBsMu9INMfwAMt48Io0wm3wmbklVHogWWmqpcG3HPiFzDUGbWdhk+mxtlCTjIDD9M0oCFzjo7X5X9wZK0TbJD7Rv9n7h4zMCwJcok5z+VFLPsv5hxNAgos1tiEp8W2pVIKZ1vk3FOjTeijzo1/PGbeMeUJwUbswvX+ZIcUK00fS2K4ujLNWy66OEAl+YkIuRoembK7hpHRxFvxBg3Kl/PU5drEewDlg7xP/H1jR+7WVTko0xvwyidQx7P1UL7E1f X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: folio_add_file_rmap_range() allows to add pte mapping to a specific range of file folio. Comparing to page_add_file_rmap(), it batched updates __lruvec_stat for large folio. Signed-off-by: Yin Fengwei --- include/linux/rmap.h | 2 ++ mm/rmap.c | 66 ++++++++++++++++++++++++++++++++++---------- 2 files changed, 54 insertions(+), 14 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index a4570da03e58..974124b41fee 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -198,6 +198,8 @@ void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_add_file_rmap_range(struct folio *, unsigned long start, + unsigned int nr_pages, struct vm_area_struct *, bool compound); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); diff --git a/mm/rmap.c b/mm/rmap.c index 8287f2cc327d..c07c4eef3df2 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1303,31 +1303,44 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, } /** - * page_add_file_rmap - add pte mapping to a file page - * @page: the page to add the mapping to + * folio_add_file_rmap_range - add pte mapping to page range of a folio + * @folio: The folio to add the mapping to + * @start: The first page number in folio + * @nr_pages: The number of pages which will be mapped * @vma: the vm area in which the mapping is added * @compound: charge the page as compound or small page * + * The page range of folio is defined by [first_page, first_page + nr_pages) + * * The caller needs to hold the pte lock. */ -void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) +void folio_add_file_rmap_range(struct folio *folio, unsigned long start, + unsigned int nr_pages, struct vm_area_struct *vma, + bool compound) { - struct folio *folio = page_folio(page); atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool first; + unsigned int nr_pmdmapped = 0, first; + int nr = 0; - VM_BUG_ON_PAGE(compound && !PageTransHuge(page), page); + VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); /* Is page being mapped by PTE? Is this its first map to be added? */ if (likely(!compound)) { - first = atomic_inc_and_test(&page->_mapcount); - nr = first; - if (first && folio_test_large(folio)) { - nr = atomic_inc_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } + struct page *page = folio_page(folio, start); + + nr_pages = min_t(unsigned int, nr_pages, + folio_nr_pages(folio) - start); + + do { + first = atomic_inc_and_test(&page->_mapcount); + if (first && folio_test_large(folio)) { + first = atomic_inc_return_relaxed(mapped); + first = (nr < COMPOUND_MAPPED); + } + + if (first) + nr++; + } while (page++, --nr_pages > 0); } else if (folio_test_pmd_mappable(folio)) { /* That test is redundant: it's for safety or to optimize out */ @@ -1356,6 +1369,31 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, mlock_vma_folio(folio, vma, compound); } +/** + * page_add_file_rmap - add pte mapping to a file page + * @page: the page to add the mapping to + * @vma: the vm area in which the mapping is added + * @compound: charge the page as compound or small page + * + * The caller needs to hold the pte lock. + */ +void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, + bool compound) +{ + struct folio *folio = page_folio(page); + unsigned int nr_pages; + + VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); + + if (likely(!compound)) + nr_pages = 1; + else + nr_pages = folio_nr_pages(folio); + + folio_add_file_rmap_range(folio, folio_page_idx(folio, page), + nr_pages, vma, compound); +} + /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from From patchwork Mon Feb 6 14:06:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13129997 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CEC5C61DA4 for ; Mon, 6 Feb 2023 14:08:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 103106B0075; Mon, 6 Feb 2023 09:08:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0644F8E0001; Mon, 6 Feb 2023 09:08:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6EA86B007B; Mon, 6 Feb 2023 09:08:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D0A5D6B0075 for ; Mon, 6 Feb 2023 09:08:39 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5ED4E1405FB for ; Mon, 6 Feb 2023 14:08:39 +0000 (UTC) X-FDA: 80437047558.26.C0777B5 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf05.hostedemail.com (Postfix) with ESMTP id 55C66100010 for ; Mon, 6 Feb 2023 14:08:37 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=JBJ436BT; spf=pass (imf05.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675692517; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pI2YIm9uTJ/8F8zs14anm/Sl2oGn/9N/fgMyadsfGUY=; b=ulhzSx0EpSFsb9vTFwzahwOL+nddcsULZ3GSds26Y7fTHcIN+CfwObplW+pa8ebP1pLJG9 yRLPQ7do894OndMiwWJSPQ3G9tvD0Fs8ydQfvyVJyDJ5pkDLxd/xGQf19isDjCGEMNoFK5 lszWagCX1C1ePGJdbOnjLdTocVTeQmM= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=JBJ436BT; spf=pass (imf05.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675692517; a=rsa-sha256; cv=none; b=Mq/yvDi0/vkIO8l0WGeYRT4AdPh04u53JOAPdSLFiSTzpya+uLUrRm2XJIcRE5Rnro36nm Ll2GzwTpn2UcS6F0rxd68wVMh3qkplduFwXYi/+XUBnxsssCMfC9fyLJxpuqel1eKhAsjH jc3P3nsWi38ko8nqJUUT7BvCpILuPtw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675692517; x=1707228517; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OKmuIlDOdFniWnDZau9ulPpi76u0RTIHxirMDzA4a3A=; b=JBJ436BTLZ4oc6/mXv/Ph4pImKHetT0L/A9JA5r8FMxaX+3W1ACRMwof GU7N7p1mAfye3HZ/kBLVtoDCc6i+H/sCOss0tXxTmxZgS0Z8E+XwBw1DZ 82EgdzPr+B1Vw5EJaaDp/QaYEqlot+I/0smXd4hhj4TYn1aqg5uv3aSjY uTIq7zz6MK+f+dr3YUcnvx79yG0aDv4wWOM6z+zmDla2OKp4k5O1qjyXG 3FotOgqqWln6SmMKiSXZonlJRflrgn9gUK/ALxwHf9LCZGnTKJ8CAwxCb waF2AqIVU+APvR587GwinCoF1rt657WseCvZ25nwUaVKmpHSLJVAMMt7U w==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="312864285" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="312864285" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Feb 2023 06:04:54 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="911937888" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="911937888" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga006.fm.intel.com with ESMTP; 06 Feb 2023 06:04:53 -0800 From: Yin Fengwei To: willy@infradead.org, david@redhat.com, linux-mm@kvack.org Cc: dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, fengwei.yin@intel.com Subject: [RFC PATCH v4 3/4] mm: add do_set_pte_range() Date: Mon, 6 Feb 2023 22:06:38 +0800 Message-Id: <20230206140639.538867-4-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230206140639.538867-1-fengwei.yin@intel.com> References: <20230206140639.538867-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 55C66100010 X-Rspam-User: X-Stat-Signature: tiep83s5cxsm3tt4mqg8o9nzmecxty1x X-HE-Tag: 1675692517-598466 X-HE-Meta: U2FsdGVkX1/VXPVnRS2gb0qkrK+0QYfQuCzwKmi4xV1zHBx+dZpdjLqT9Sb9A1Hhnw6d8ijfdVe2A1fbGXrXjY0ajg/+464kJGAetLq4JdDDAsej/zBHkr9P8fnnbTO5gDqTR1ng9QujzbeWVMaXQIettEYwlRiZlz2w7xAhwBqsvXi2IJ+rEB3dTWsLs1ileyVl386kyDnKVYIXZcpz2ZTx2PMlFeXlvHQ3oo7VMwlfD9Jipv5R//BsaiOvQ9UKiHCTV/zHc0mZZgfoa5vUq591QWWCdszcA9auI7ZaQfrOZ/JrHvA14ubLzZOWohFvExX2kPUX+DHVqroGoHkG1WQ5kCYVVZUOuNFVu6tmQNOdGM1LSbof29lYGXmHUVNEzdGwDW7lN9eJDRx8AZPgxCmvwOq/YBMZpspEiWwogB4eYVs8Sx8Pg4iJvlpORrAppJjDlVq2G2cVlyNyQvWHYnIoaTQtd7sOfXW+98DddbSOdHaA3BL4QpyjaEVTKRylySin1VfAl4cGhN5AV6xJ8xxbhufB0S0DpJFBEHdjF3HMbdTCbWQMigUfrJCyWj4QGSLl73EqmKfAfK5u384m9XInxCSxOD35dRe7uTxGODJHwWm8y5Gy/1WN1RQGGoEl6Wi51IXZ1VMbJmGb4WZX5vIPPBuN6TAiqxVJKTVQEKxwvas0r7IF82TuJQ8Z0WvxgCrDIQNvhSobt23PN0foFab+jT8bN3mfmZVenlpdw0BJ/jl2WBP1x2S9ItP7lXNSE4SHPH0IMK5r8GtxBbAcrtzjNuORMVnsgDGpU2X6IgifO9CenNAIzqOqM6/GRmj65B0YA4FdI89keFqu0/ebsYGZF3IPSVL0qFKNwMx8CNcGvvc1GeTXQQeY9vGXNgz5VauV6mLlHo1v6tYJ0E4eVfzeW535aznAnjcKUNHsBPOAMLaOjR94gBx7FmD5tBQKDZUsQD8GeNSyaNTuTnv Rikxi0E7 iA0/zjqmewz57tdBFEYdrQ9EwSh14YAESjfCTCyv+Ra2Q4hy2oPSPgzCqaWPjE4ZUoGlbJT7eqnZVSRTPK1swzrTgLodk9NuegJ9bDug0ewy21kqF4tnP6sRnEZjKcWK8UuCwTBTLl0l4f8XU8bKhVsA/jetlDJ7L82Ya95hbo0HM8drrr+sVH9Ov5rSJRWb7oqQuoZsVZCQvU+XCsXuoqmwSRQhCjzDpAAEu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: do_set_pte_range() allows to setup page table entries for a specific range. It calls folio_add_file_rmap_range() to take advantage of batched rmap update for large folio. Signed-off-by: Yin Fengwei --- include/linux/mm.h | 3 +++ mm/filemap.c | 1 - mm/memory.c | 66 ++++++++++++++++++++++++++++++++-------------- 3 files changed, 49 insertions(+), 21 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d6f8f41514cc..93192f04b276 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1162,6 +1162,9 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma) vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page); void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr); +void do_set_pte_range(struct vm_fault *vmf, struct folio *folio, + unsigned long addr, pte_t *pte, + unsigned long start, unsigned int nr); vm_fault_t finish_fault(struct vm_fault *vmf); vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf); diff --git a/mm/filemap.c b/mm/filemap.c index 1c37376fc8d5..6f110b9e5d27 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3376,7 +3376,6 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, ref_count++; do_set_pte(vmf, page, addr); - update_mmu_cache(vma, addr, vmf->pte); } while (vmf->pte++, page++, addr += PAGE_SIZE, ++count < nr_pages); /* Restore the vmf->pte */ diff --git a/mm/memory.c b/mm/memory.c index 7a04a1130ec1..51f8bd91d9f0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4257,36 +4257,65 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) } #endif -void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr) +void do_set_pte_range(struct vm_fault *vmf, struct folio *folio, + unsigned long addr, pte_t *pte, + unsigned long start, unsigned int nr) { struct vm_area_struct *vma = vmf->vma; bool uffd_wp = pte_marker_uffd_wp(vmf->orig_pte); bool write = vmf->flags & FAULT_FLAG_WRITE; + bool cow = write && !(vma->vm_flags & VM_SHARED); bool prefault = vmf->address != addr; + struct page *page = folio_page(folio, start); pte_t entry; - flush_icache_page(vma, page); - entry = mk_pte(page, vma->vm_page_prot); + if (!cow) { + folio_add_file_rmap_range(folio, start, nr, vma, false); + add_mm_counter(vma->vm_mm, mm_counter_file(page), nr); + } else { + /* + * rmap code is not ready to handle COW with anonymous + * large folio yet. Capture and warn if large folio + * is given. + */ + VM_WARN_ON_FOLIO(folio_test_large(folio), folio); + } - if (prefault && arch_wants_old_prefaulted_pte()) - entry = pte_mkold(entry); - else - entry = pte_sw_mkyoung(entry); + do { + flush_icache_page(vma, page); + entry = mk_pte(page, vma->vm_page_prot); - if (write) - entry = maybe_mkwrite(pte_mkdirty(entry), vma); - if (unlikely(uffd_wp)) - entry = pte_mkuffd_wp(entry); - /* copy-on-write page */ - if (write && !(vma->vm_flags & VM_SHARED)) { + if (prefault && arch_wants_old_prefaulted_pte()) + entry = pte_mkold(entry); + else + entry = pte_sw_mkyoung(entry); + + if (write) + entry = maybe_mkwrite(pte_mkdirty(entry), vma); + if (unlikely(uffd_wp)) + entry = pte_mkuffd_wp(entry); + set_pte_at(vma->vm_mm, addr, pte, entry); + + /* no need to invalidate: a not-present page won't be cached */ + update_mmu_cache(vma, addr, pte); + } while (pte++, page++, addr += PAGE_SIZE, --nr > 0); +} + +void do_set_pte(struct vm_fault *vmf, struct page *page, unsigned long addr) +{ + struct folio *folio = page_folio(page); + struct vm_area_struct *vma = vmf->vma; + bool cow = (vmf->flags & FAULT_FLAG_WRITE) && + !(vma->vm_flags & VM_SHARED); + + if (cow) { inc_mm_counter(vma->vm_mm, MM_ANONPAGES); page_add_new_anon_rmap(page, vma, addr); lru_cache_add_inactive_or_unevictable(page, vma); - } else { - inc_mm_counter(vma->vm_mm, mm_counter_file(page)); - page_add_file_rmap(page, vma, false); } - set_pte_at(vma->vm_mm, addr, vmf->pte, entry); + + do_set_pte_range(vmf, folio, addr, vmf->pte, + folio_page_idx(folio, page), 1); } static bool vmf_pte_changed(struct vm_fault *vmf) @@ -4361,9 +4390,6 @@ vm_fault_t finish_fault(struct vm_fault *vmf) if (likely(!vmf_pte_changed(vmf))) { do_set_pte(vmf, page, vmf->address); - /* no need to invalidate: a not-present page won't be cached */ - update_mmu_cache(vma, vmf->address, vmf->pte); - ret = 0; } else { update_mmu_tlb(vma, vmf->address, vmf->pte); From patchwork Mon Feb 6 14:06:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13129998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E922EC05027 for ; Mon, 6 Feb 2023 14:08:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56E766B0078; Mon, 6 Feb 2023 09:08:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 51E4B6B007B; Mon, 6 Feb 2023 09:08:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 320586B007D; Mon, 6 Feb 2023 09:08:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1121E6B0078 for ; Mon, 6 Feb 2023 09:08:40 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8B38FA8F83 for ; Mon, 6 Feb 2023 14:08:39 +0000 (UTC) X-FDA: 80437047558.25.83A3D58 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf22.hostedemail.com (Postfix) with ESMTP id 59141C0022 for ; Mon, 6 Feb 2023 14:08:37 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=YAxb32ob; spf=pass (imf22.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675692517; a=rsa-sha256; cv=none; b=GkreBIRpCWEiswPZ1VnTDhJ35h9wqBc764JVKmTjpgbIJEZNmmbIvG0dW45Ly67sI7oiA1 g1OG/9/fPHaKX529fpXOW5YRd7GwIR3i4whJXvemNPF/ddUL+RhG+tCKX/TZ+41rYNwSxv hhxdBPnFI4/wRlbCXRZ66ZU4O0CMJss= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=YAxb32ob; spf=pass (imf22.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675692517; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FnDsIyq2O3ERrsvZTiqg/uX15dZjf3zoty8Xi+r86uY=; b=z9gRTotiwzJNhFZwMeAvuP8Rkn6z8owAsopdejpPXQoxPAwsHLaTm4T2dXDC7wq2AmyDhq d8EUl6UMBy44gZ5sCBs2gJTGyKIs0RZ4NE6zjHXhomhY+M6sLGaGtQeCtVoQyJ0w1ROgz1 TP3vw3VLAu1UrbH7SBXWEq8Ag8Ip1fU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1675692517; x=1707228517; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=U1Eiy12DbqHzX/ZOo41S8PKwsnJTE57YtWWPprWmJGw=; b=YAxb32obP/u0RZ83HFGmbu93K2cfi2Fi4OxKWeCYALjEg4c7KHjkLLjD ZOlY9KaDdkIjuemrFFRWMrbKPseleW98iP+cXjWdVlj4k7JbO2xvAfzY6 u/+81Tn3R6s0F0CcVgzeEDtOV4Reo1vgjmGZP8a97BoZuR/G4pbfGRyWi gyMNRP3lrcBxr0huXrkPfzV1HD3wqSAoWUxnhmksoeHF9ywjWUipt62v+ ONabIPKqYtIEv3pmESRKWl5llZfIE9cC+MCw3Uaw9sGz+Me1lIb/BM2j8 jqFrcGEBvCVwcDSrctxhFVTnuvRyxzGUDWyNHi6BinkASa37rcI5zSf+H w==; X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="312864312" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="312864312" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Feb 2023 06:04:57 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10612"; a="911937897" X-IronPort-AV: E=Sophos;i="5.97,276,1669104000"; d="scan'208";a="911937897" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga006.fm.intel.com with ESMTP; 06 Feb 2023 06:04:55 -0800 From: Yin Fengwei To: willy@infradead.org, david@redhat.com, linux-mm@kvack.org Cc: dave.hansen@intel.com, tim.c.chen@intel.com, ying.huang@intel.com, fengwei.yin@intel.com Subject: [RFC PATCH v4 4/4] filemap: batched update mm counter,rmap when map file folio Date: Mon, 6 Feb 2023 22:06:39 +0800 Message-Id: <20230206140639.538867-5-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230206140639.538867-1-fengwei.yin@intel.com> References: <20230206140639.538867-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 59141C0022 X-Rspamd-Server: rspam01 X-Stat-Signature: i5xabk6r5e7meihjw4mp1o9uncpiqx9q X-HE-Tag: 1675692517-101089 X-HE-Meta: U2FsdGVkX1/CKED5oslysRdMEB4knFHNIqVaMEp9Kmgl+7N3y0rVcHDmVjiabtA0zGCH1VWqNkIJKwR/aIUxmUNo8W8zPesVkMavVMXcG74BFNdHJw/7/Bz0+g8JHMEIxLYxBfhQTaR3H9ucNhDcvJel4qcLHWtg+0Etde3K5UkuauxwamBscp9l/soomLIHl00YOTI/MWdGzSxXS1Id0oHeOgXkGn5cEJrMq40qCfueS3v5FEI07Hvz1HJiUoAA99MLTF7gcLNXF9Pf35F5adG70or6wMzA1WTJg9ID5shC2ij0KtWr1Z8EjXoQbpM9EeUPYE8DOAvzL1TGUqOPp6PbSPF+XPaX8N+58vLfFYRdNIlMHVWF68ips6pk5AWQTyuEKjbGizXNhXqxLCXp+3FQ2G1zPu3MBYtPWxVJ2AAl0PZ5uvz+ko9OsVrh81RF28Y4HTnWJkPUKLR2UXyX/rYQVNRrxbJ72O7C5mIGbm1Ymg8sFeV8OdQZhF3YQ3ZZYrDqw16Q4vqr3T1QWNse/AmRooRGPA0dHYmouSx+veps2ZRRjjWp3Myg5b/SfJ4gYOjVvmXGhsZema7TlLielwPWoMv1ij+d6JXbQMQQDpGUUIiJHVqj/wthyOVizz+jPHji6u5fLD8QBsyoPB2cSXt4SeepsbAL4kUpremHb4hIdr+JdfciieZzQjQh/fanfBhYXvOYt2SY4t6RR5F4x5Ktg6ED4LGOreCdQV6yduA6r1OwgOgRY6lsjiSabwhwnEwkRvp32bO0otV4aNJyiMusLrvnO79/OOfWca2NKJ7FIYuFSuqcBwMEcEjtc55CWY28hngbwTKaftm1UZ88HFCIUO8FxUSXxSWR+Fyf/zGB5BkwwZwCRXlGDfsJ8svOT0T/BTnSaEPMr30oaY4FGiTqlpFHgvnAoBwncuv9rpqWoorv0ER+zRJMh+ML7jK0E+zgbXHV+t4Zat8bfxT b7ymlMAY hlc5Fb22nvCxpIf/XjNtvkAW0RS4O6dHmEcOhWO8IoRjBuf8pwCpC67hbYdHDIUK2/Z2NDx9KVe82UDYb1S5g6DsBjMEvApBIBizetXKE6R5UYv8HySe0J3aMjUrVt3lIVtPtS0VtucpBcsDFzIgKOWFQXQtM6xjpMA4FHD66JtRlqIs0TNYGmgqrC94Gu7fXM6MzPwnVceYDNBI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use do_set_pte_range() in filemap_map_folio_range(). Which batched updates mm counter, rmap for large folio. With a will-it-scale.page_fault3 like app (change file write fault testing to read fault testing. Trying to upstream it to will-it-scale at [1]) got 15% performance gain on a 48C/96T Cascade Lake test box with 96 processes running against xfs. Perf data collected before/after the change: 18.73%--page_add_file_rmap | --11.60%--__mod_lruvec_page_state | |--7.40%--__mod_memcg_lruvec_state | | | --5.58%--cgroup_rstat_updated | --2.53%--__mod_lruvec_state | --1.48%--__mod_node_page_state 9.93%--page_add_file_rmap_range | --2.67%--__mod_lruvec_page_state | |--1.95%--__mod_memcg_lruvec_state | | | --1.57%--cgroup_rstat_updated | --0.61%--__mod_lruvec_state | --0.54%--__mod_node_page_state The running time of __mode_lruvec_page_state() is reduced about 9%. [1]: https://github.com/antonblanchard/will-it-scale/pull/37 Signed-off-by: Yin Fengwei --- mm/filemap.c | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 6f110b9e5d27..4452361e8858 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3354,11 +3354,12 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, struct file *file = vma->vm_file; struct page *page = folio_page(folio, start); unsigned int mmap_miss = READ_ONCE(file->f_ra.mmap_miss); - unsigned int ref_count = 0, count = 0; + unsigned int mapped = 0; + pte_t *pte = vmf->pte; do { if (PageHWPoison(page)) - continue; + goto map; if (mmap_miss > 0) mmap_miss--; @@ -3368,20 +3369,34 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf, * handled in the specific fault path, and it'll prohibit the * fault-around logic. */ - if (!pte_none(*vmf->pte)) - continue; + if (!pte_none(pte[mapped])) + goto map; if (vmf->address == addr) ret = VM_FAULT_NOPAGE; - ref_count++; - do_set_pte(vmf, page, addr); - } while (vmf->pte++, page++, addr += PAGE_SIZE, ++count < nr_pages); + mapped++; + continue; - /* Restore the vmf->pte */ - vmf->pte -= nr_pages; +map: + if (mapped) { + do_set_pte_range(vmf, folio, addr, pte, start, mapped); + folio_ref_add(folio, mapped); + } + + /* advance 1 to jump over the HWPoison or !pte_none entry */ + mapped++; + start += mapped; + pte += mapped; + addr += mapped * PAGE_SIZE; + mapped = 0; + } while (page++, --nr_pages > 0); + + if (mapped) { + do_set_pte_range(vmf, folio, addr, pte, start, mapped); + folio_ref_add(folio, mapped); + } - folio_ref_add(folio, ref_count); WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss); return ret;