From patchwork Fri Jul 28 07:09:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13331302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10689EB64DD for ; Fri, 28 Jul 2023 07:11:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 90E9A6B0074; Fri, 28 Jul 2023 03:11:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8BCF96B0075; Fri, 28 Jul 2023 03:11:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7837A6B0078; Fri, 28 Jul 2023 03:11:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6D9EA6B0074 for ; Fri, 28 Jul 2023 03:11:17 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3FDC7C0889 for ; Fri, 28 Jul 2023 07:11:17 +0000 (UTC) X-FDA: 81060149394.07.10068E1 Received: from mgamail.intel.com (unknown [192.55.52.43]) by imf23.hostedemail.com (Postfix) with ESMTP id EC411140006 for ; Fri, 28 Jul 2023 07:11:14 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=RWFL4d7x; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf23.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690528275; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ndav55EIknyWAhlk4Z6h8jE2vJe9nExG9vzBr9pdpiA=; b=F/ShvIhbePr7/6RYPDtDDa/sba8TpmILzxDUHITCrbWXfjqqHWyk7Ru7AZfz452S8vdNBb Z0ruI4Ksce4XZIngGwrjBNJPIWBQcuOjFBsK6hyVXu8+1soKBbMRmYKajzarMYgpbo0v4b 2WezwIgLgsE+oBqVvOZLLv/XSUUXY/4= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=RWFL4d7x; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf23.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690528275; a=rsa-sha256; cv=none; b=z6NVFvZ4T6eJ+rqLQkim6NiTm4ptYtdWN7hxNlS9aYyZTjVLE50Y4Sc4uBjSrvAOntAO3J ycaPxJ2evLnTWVcru9660xKFG+3auo7JfmoqWd0ezQCwnKba5loX/ZUAzENvP5v1o44k5X swuBy7zZU+lAzBf4oTBYtx51tOUT2v0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690528275; x=1722064275; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hDGtd2tgXIan13dy2PEKZ68DXcw4gSjf8pK8pQRP1gI=; b=RWFL4d7xTcNVQnXbaM1WYUb1S8Plz3HF6NtHedw076tQMiysJ8NVdiMQ Ipm2woSe8e2mF08wdMw1IcQi72XwnZikmyw7n0jmj3jNhoAcRE2LUygDF +oSAe0LnOx+C1zDZDhO3JGwLOrIOQXZhPyWUBnDENBh8mEzf2jv4Nd7mf uZH4UGXisTklyXHp4MoZdi9k30lnWJPMO0YZYy65opP13ppXsHuaMa+O7 MsEM5K/UfHtZkoZeA1NMCqmJc2Ow7skEpRDnK0yOnO2QCXfMVXugjfJtR jSJ/9tlIp1519UBvw/Fuk+1r8dQ3/vuFXVmO7WCeF+A3pugq9+5ra8wZH A==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="454888120" X-IronPort-AV: E=Sophos;i="6.01,236,1684825200"; d="scan'208";a="454888120" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jul 2023 00:11:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="721161998" X-IronPort-AV: E=Sophos;i="6.01,236,1684825200"; d="scan'208";a="721161998" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga007.jf.intel.com with ESMTP; 28 Jul 2023 00:11:08 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, yuzhao@google.com, willy@infradead.org, david@redhat.com, ryan.roberts@arm.com, shy828301@gmail.com, hughd@google.com Cc: fengwei.yin@intel.com Subject: [PATCH 1/3] mm: add functions folio_in_range() and folio_within_vma() Date: Fri, 28 Jul 2023 15:09:27 +0800 Message-Id: <20230728070929.2487065-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230728070929.2487065-1-fengwei.yin@intel.com> References: <20230728070929.2487065-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: EC411140006 X-Stat-Signature: m4f516f6s5k6fmwirxesjdkhqcdjsb7i X-HE-Tag: 1690528274-8556 X-HE-Meta: U2FsdGVkX18DE9Los2E1IlRRHeXvzY5sFTPiEKCEefz34cI2hEgBLaH0lEt9HOcz8JLyk8RT4zkAUZEEHophK7cbNolZ5q5q49AZEii7zuz/91kOkuqvWcbd2IO+RTNvHozT5UK7xcTLMlnhrGi+WaZ3XySLl/iSRW4RYkwLmSebSeVS0/jWMrVQ0Kgwzy/Cukhw76hv/k0NG7J6YvAd8DidSNf5elYgHxQhT+DY25UEjqdunQtdtCoVLwiCQu1v7tP4hU8fC+6F1OfNfqLG8XNjk2Cd1Tw1NmkIzlaZo+x9hULaFFUM7goE9s6/B3v7xA64cC8BJpsOEyyysejzIrr6V/41FcqOa/ZrFdwmgp1t1guFgFSuVcJcc4GnFwNYwaVxTL8ttZE9CRXiSp/od/9a0wxlfT/YKp3ujX+8lpxYuy7vptFyMnoNSZCsI7svD0hXlmttjPYtSUse2Fc51j0E7X9T+nz/WIx1ocpsMbVbEo6+LjwpCj0rzdgD1zFhO6xID6VBCV4WA8o/ngwxeXHVqO+XRTgwF+XHen01B/ejl3LChjYRtTH7l53JZ0xV3xru4DmUmNUXCJX3EA/C5u9Aloea/7vcZGnOtZ0KR1dQs5HUbb9GPEyLc9WSkVrNrXCc8gY5Gq50SehYNcHNndDU5TtpJ5loq0c71ogf97zIVuaJhVH59mnArwzHlobxY80KZr/KVnIR1+Hpwmwb2ZnvapCmnhNGdAtUlnOmtF47WAGHaiJtmPGhon34Uf9WbkhW1YC6M24pEsem5s28o2wuknXM+CPC8Q14TECEA3XIj5poNjKSVGjKwb2UfO+cJSBFzm+mDDHcmX1+pAG5rVHGKgQb4v4vIdNWaNjnVUt9vesEuJCK/qEucYPvSeInDA6tFfKfSsFGu32aNGgZcNlA0epaL9sSwX85CwbwCfEtIZT5ybqbxTzAA8Oclh1AS66w4s0j/j/rBvE7UU0 j5V9HI0s gmdKNjdhYQ8KCTQdrVt3cOWWnWqWER2ZVQIcWFlERVAJQEFSLPEJ3p2jjRQfdFw+cd5k90CahfjPKf2Ltt08uMUMR55DaG10DUMF9YQX/o84T5TyWwswMAoqiRslP/hNHjniycaRbWtGEAGI/S3DIVX8MB4nDLdhPAmTML8HPRHMIj8oSsmFqrmLLRQBNDw+2X+/NhUsbeT2xChUjQvLVXqSiZMoGKH/0kwfedEusxrxdUuHw3gwKwkNNlQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It will be used to check whether the folio is mapped to specific VMA and whether the mapping address of folio is in the range. Also a helper function folio_within_vma() to check whether folio is in the range of vma based on folio_in_range(). Signed-off-by: Yin Fengwei --- mm/internal.h | 69 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/mm/internal.h b/mm/internal.h index 5a03bc4782a2..63de32154a48 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -585,6 +585,75 @@ extern long faultin_vma_page_range(struct vm_area_struct *vma, bool write, int *locked); extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags, unsigned long bytes); + +/* + * Check whether the folio is in specific range + * + * First, check whether the folio is in the range of vma. + * Then, check whether the folio is mapped to the range of [start, end]. + * In the end, check whether the folio is fully mapped to the range. + * + * @pte page table pointer will be checked whether the large folio + * is fully mapped to. Currently, if mremap in the middle of + * large folio, the large folio could be mapped to to different + * VMA and address check can't identify this situation. + */ +static inline bool +folio_in_range(struct folio *folio, struct vm_area_struct *vma, + unsigned long start, unsigned long end, pte_t *pte) +{ + pte_t ptent; + unsigned long i, nr = folio_nr_pages(folio); + pgoff_t pgoff, addr; + unsigned long vma_pglen = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; + + VM_WARN_ON_FOLIO(folio_test_ksm(folio), folio); + + if (start < vma->vm_start) + start = vma->vm_start; + if (end > vma->vm_end) + end = vma->vm_end; + + pgoff = folio_pgoff(folio); + /* if folio start address is not in vma range */ + if (pgoff < vma->vm_pgoff || pgoff > vma->vm_pgoff + vma_pglen) + return false; + + addr = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); + if (addr < start || end - addr < folio_size(folio)) + return false; + + /* not necessary to check pte for none large folio */ + if (!folio_test_large(folio)) + return true; + + if (!pte) + return false; + + /* check whether parameter pte is associated with folio */ + ptent = ptep_get(pte); + if (pte_none(ptent) || !pte_present(ptent) || + pte_pfn(ptent) - folio_pfn(folio) >= nr) + return false; + + pte -= pte_pfn(ptent) - folio_pfn(folio); + for (i = 0; i < nr; i++, pte++) { + ptent = ptep_get(pte); + + if (pte_none(ptent) || !pte_present(ptent) || + pte_pfn(ptent) - folio_pfn(folio) >= nr) + return false; + } + + return true; +} + +static inline bool +folio_within_vma(struct folio *folio, struct vm_area_struct *vma, pte_t *pte) +{ + return folio_in_range(folio, vma, vma->vm_start, vma->vm_end, pte); +} + /* * mlock_vma_folio() and munlock_vma_folio(): * should be called with vma's mmap_lock held for read or write, From patchwork Fri Jul 28 07:09:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13331303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8CD8EB64DD for ; Fri, 28 Jul 2023 07:11:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 562BD6B0075; Fri, 28 Jul 2023 03:11:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C4856B0078; Fri, 28 Jul 2023 03:11:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33F666B007B; Fri, 28 Jul 2023 03:11:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 22D186B0075 for ; Fri, 28 Jul 2023 03:11:29 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DC096120FBF for ; Fri, 28 Jul 2023 07:11:28 +0000 (UTC) X-FDA: 81060149856.06.F4ABF79 Received: from mgamail.intel.com (unknown [192.55.52.43]) by imf26.hostedemail.com (Postfix) with ESMTP id 870CF140017 for ; Fri, 28 Jul 2023 07:11:26 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QoA93gDN; spf=pass (imf26.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690528287; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EePU6xPznB5l9WtFwzndYBUAVWzLnMkN9vq47y9+zRY=; b=IA+b+szlgy+0Zq/Nw7LckWIv13wD7GY3ze/POyLRtEF6BuTbOoywfSulbwDfea/1Xb++Gn 0nytNYPn1X0dlgUTEHd1HdTeA3F/ytF6aWiZVylCtQcPGVLPMrgF+HSiHNnBCuMS/biJM/ Toml2oDu/Vr0cMpoG9PEd+Pk6AVXO0k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690528287; a=rsa-sha256; cv=none; b=3cP8TYWdkQ0xpNU/N4AWuDzuzGxs8fMp6yJH2qVVYxEN9aV1EX7X3RbMON/OP0f8ncl2Rz lFv69r0HQ4JJjSryy0JIAmEapxIj2klXxw42shmHnmrJno/PyAWdckvSdII8d4MdCaj2sm H7uu4zgOyB65PzVx9H9k9b9MTAnUv+A= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=QoA93gDN; spf=pass (imf26.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690528286; x=1722064286; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hSd/zYXrNe6G/TyNoyJX1Y9vNt38Z30KKrdYuOwbs5U=; b=QoA93gDNTs2R27Q9OvJtqUI5bcyZGejODNep5nWoHIpOnKE8Tgc3do26 nzvkfCkmtOaJawScGXKyPNXkyU/dkrbZn8Q4jgjkT1dFF/v8nXoJY0Ivf jdDI08ofx9gpfpi766gShtJaMT7L9SvL4j7bufxE12Yj/hKpt8vDUcLdk 6S3ygnqlBGdz7lR3IzgJ2Li5TmBntRdcqOLfbtkRkmcSkHlRCfTn2u6dh GB2S3OS5RoRgA4YC7hW0fkUyXS/9X90qjtk0BSDoJ8dlTWnJ6Vnbwyrci +wWuqRZawjZB7lIFd85crssdUAtV/638ryokknhlDp94+PQXuvZeZ2FX8 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="454888197" X-IronPort-AV: E=Sophos;i="6.01,236,1684825200"; d="scan'208";a="454888197" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jul 2023 00:11:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="721162102" X-IronPort-AV: E=Sophos;i="6.01,236,1684825200"; d="scan'208";a="721162102" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga007.jf.intel.com with ESMTP; 28 Jul 2023 00:11:22 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, yuzhao@google.com, willy@infradead.org, david@redhat.com, ryan.roberts@arm.com, shy828301@gmail.com, hughd@google.com Cc: fengwei.yin@intel.com Subject: [PATCH 2/3] mm: handle large folio when large folio in VM_LOCKED VMA range Date: Fri, 28 Jul 2023 15:09:28 +0800 Message-Id: <20230728070929.2487065-3-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230728070929.2487065-1-fengwei.yin@intel.com> References: <20230728070929.2487065-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 870CF140017 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 9b7bkzso97gom43k7djhk4be5rc9t6xu X-HE-Tag: 1690528286-412005 X-HE-Meta: U2FsdGVkX1+iyiV95pNCYGIYM9KTEnAQBRYGRSRAqe+lInUO+7ajDKSwLQryeK15HYJavJnZKuPXzB1pgfOxPtGgu5uoZROxttB7Fn8ZW/SB0m+Nro5FGkXfeQwA9YeTzidfrTqNVbGXG2BHgTIF6LhMg0jRatuO3mRiC4RJG/8hR0PTURxnWFhoGMhNE62deSUWkjx3xSgHlNCY2efmqnT5cwrJ2SP4CzFErD/uz7UyuamXTlIAYmshzlat+GW1b3Y0Gcng2NgRDjX3EAF93Gfmf6ir4SkVDdjFA/My39VFYejBPxh6n7IPZAxGkmMOPxB9tNseaFyvErlvQ5D6KCE5OAbwPxwJ15GXwbWiQWqOR4pyFqEj7toztWdk3+m5oFqNIftGU8jY+pGL5eKG+j72MgyKpotdsyMQ/h66DZCDskq+SP6/KwE/TRlCjvp4kWZacaIedPXlHilJ7Tesn/ms+dummRmGxb1vbCo8CV4RbLRDhiEP3K9Yha/OBIuMHsJ84/VFLfBxxfmeJmmQKwl9J3tb9ew8RmOG0JglffMl4LoKf5eSgONBI1BpgYrtQGmXt/sWgcPClAS9mi5IMGbui2kUwq1qCWQlKZtewvgu3UAZjGW2IJiweFqRIKoeTPohvnip2xYW1qmXTos5/Jlb8h4tG3sgCFjGX24PB5Moth4D1vqsQGlhT72Ag9SAJ8VC7L1ds4jdjFFsS9fMHiEitKW84paf7/KtKgG7FYL4phD3wgs1fm/ow6D02Ggghtm1GsYFVLHzewpYu6MlJBqK590esK2gToUodfIyIi5IP82mqLLRMtRcLmRT7G/N1aGvGO5O+FzRJ9QKq+O/DxpASWe7EYxp1Zlp4OgleK7WkbAQPo+HVsovq/FMZVSO8OQ3qeqCkmLeCE5XrR61wgkc32FFVDmurIiUD7CcpK1DvM8CidIu13VWELgthoFUHDuVyDQinl0dF86TgnX vV6sg5Ja nT6viJPcVm7EuKE1k51+QvFQCzmNuHmfAoQoUtj/pdMk72NQVw/IlBqB6oT98A86tNkDFLDl+oKsHNWftNKXFxs1korlOcuSrU8nBEL14RT71cpawXAHls8mRbhhfdu5w+wwx/DQCtIHbtzEDjz4LeKD3p3prK78fHzsWA1+74R5kWS+VbOBOcBa9U0Cbg9CsctL4YIv3nr9M8KMKga4dZ03tKzFbFgcG0+fBTOlcDGCmygigo+IEoNsxaQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If large folio is in the range of VM_LOCKED VMA, it should be mlocked to avoid being picked by page reclaim. Which may split the large folio and then mlock each pages again. Mlock this kind of large folio to prevent them being picked by page reclaim. For the large folio which cross the boundary of VM_LOCKED VMA, we'd better not to mlock it. So if the system is under memory pressure, this kind of large folio will be split and the pages ouf of VM_LOCKED VMA can be reclaimed. for page_add_anon_rmap() and page_add_file_rmap(), we only mlock the folio if it's not large folio. The reason to do so is that these functions can be called couple of times for a large folio and each call just covered piece of large folio. If folio is mlocked multiple time, the folio->mlock_count can be imbalance. Delay the folio mlock to page reclaim phase. As only mlock folio once for sure in page reclaim phase. Signed-off-by: Yin Fengwei --- mm/internal.h | 18 +++++++++--------- mm/rmap.c | 27 ++++++++++++++++++++------- 2 files changed, 29 insertions(+), 16 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 63de32154a48..6c6fb1f3e4c1 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -662,14 +662,10 @@ folio_within_vma(struct folio *folio, struct vm_area_struct *vma, pte_t *pte) * mlock is usually called at the end of page_add_*_rmap(), munlock at * the end of page_remove_rmap(); but new anon folios are managed by * folio_add_lru_vma() calling mlock_new_folio(). - * - * @compound is used to include pmd mappings of THPs, but filter out - * pte mappings of THPs, which cannot be consistently counted: a pte - * mapping of the THP head cannot be distinguished by the page alone. */ void mlock_folio(struct folio *folio); static inline void mlock_vma_folio(struct folio *folio, - struct vm_area_struct *vma, bool compound) + struct vm_area_struct *vma, pte_t *pte) { /* * The VM_SPECIAL check here serves two purposes. @@ -680,16 +676,20 @@ static inline void mlock_vma_folio(struct folio *folio, * still be set while VM_SPECIAL bits are added: so ignore it then. */ if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED) && - (compound || !folio_test_large(folio))) + folio_within_vma(folio, vma, pte)) mlock_folio(folio); } void munlock_folio(struct folio *folio); static inline void munlock_vma_folio(struct folio *folio, - struct vm_area_struct *vma, bool compound) + struct vm_area_struct *vma) { - if (unlikely(vma->vm_flags & VM_LOCKED) && - (compound || !folio_test_large(folio))) + /* + * To handle the case that a mlocked large folio is unmapped from VMA + * piece by piece, allow munlock the large folio which is partially + * mapped to VMA. + */ + if (unlikely(vma->vm_flags & VM_LOCKED)) munlock_folio(folio); } diff --git a/mm/rmap.c b/mm/rmap.c index 54124f18e0e4..1d8f048fbed8 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -798,6 +798,7 @@ struct folio_referenced_arg { unsigned long vm_flags; struct mem_cgroup *memcg; }; + /* * arg: folio_referenced_arg will be passed */ @@ -811,10 +812,22 @@ static bool folio_referenced_one(struct folio *folio, while (page_vma_mapped_walk(&pvmw)) { address = pvmw.address; - if ((vma->vm_flags & VM_LOCKED) && - (!folio_test_large(folio) || !pvmw.pte)) { + if (vma->vm_flags & VM_LOCKED) { + if (!folio_within_vma(folio, vma, pvmw.pte)) { + /* + * For large folio cross VMA boundaries, it's + * expected to be picked by page reclaim. But + * should skip reference of pages which are in + * the range of VM_LOCKED vma. As page reclaim + * should just count the reference of pages out + * the range of VM_LOCKED vma. + */ + pra->mapcount--; + continue; + } + /* Restore the mlock which got missed */ - mlock_vma_folio(folio, vma, !pvmw.pte); + mlock_vma_folio(folio, vma, pvmw.pte); page_vma_mapped_walk_done(&pvmw); pra->vm_flags |= VM_LOCKED; return false; /* To break the loop */ @@ -1253,7 +1266,7 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, __page_check_anon_rmap(folio, page, vma, address); } - mlock_vma_folio(folio, vma, compound); + mlock_vma_folio(folio, vma, NULL); } /** @@ -1344,7 +1357,7 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, if (nr) __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr); - mlock_vma_folio(folio, vma, compound); + mlock_vma_folio(folio, vma, NULL); } /** @@ -1383,7 +1396,7 @@ static void __remove_rmap_finish(struct folio *folio, * it's only reliable while mapped. */ - munlock_vma_folio(folio, vma, compound); + munlock_vma_folio(folio, vma); } /** @@ -1557,7 +1570,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, if (!(flags & TTU_IGNORE_MLOCK) && (vma->vm_flags & VM_LOCKED)) { /* Restore the mlock which got missed */ - mlock_vma_folio(folio, vma, false); + mlock_vma_folio(folio, vma, pvmw.pte); page_vma_mapped_walk_done(&pvmw); ret = false; break; From patchwork Fri Jul 28 07:09:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13331304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0FC8C0015E for ; Fri, 28 Jul 2023 07:11:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69C4E6B0072; Fri, 28 Jul 2023 03:11:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 623F46B0078; Fri, 28 Jul 2023 03:11:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C52B6B007B; Fri, 28 Jul 2023 03:11:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3D0D36B0072 for ; Fri, 28 Jul 2023 03:11:43 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 08DBD1A0ED6 for ; Fri, 28 Jul 2023 07:11:43 +0000 (UTC) X-FDA: 81060150486.15.5CBE854 Received: from mgamail.intel.com (unknown [192.55.52.43]) by imf14.hostedemail.com (Postfix) with ESMTP id D09B6100018 for ; Fri, 28 Jul 2023 07:11:40 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="d8/wDWU5"; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf14.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690528301; a=rsa-sha256; cv=none; b=qaVrMlVJ+fr3USIOqsaUNziwew82aD1+k9DB5e9ZG1YW9onDIEqna3A6mdJ6/uN8lmwyFZ V9NNYyVNtSpdtJDf8PC4YWGJiUT291+iVf7Us9DmtW0OKyo40VyKfiHNKdAABHoX6ZOP/P OAJY9iWRcAwDjWl4lPokCNmZmD8UgX0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="d8/wDWU5"; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf14.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690528301; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Bscpb2iXK+4PjcoQ+APmWHh3oC47C7yy4l3caB5O8Hs=; b=JLbnVvBhNvkB433/nO+yLd6e+DnPeVwy/aCOpcwnvQ03jnuIZzKyXUJTbsg4xmOV7JLUrr AkXKrziS3VC9S9UrEKkoLKmKzA4UV7Xm66xyZkFOXaff1w0BtJL2AB5Owk77/xtc1JBq5Q eXZoJ47F5ZQPC7PuWcfGSYxamK2ED24= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690528300; x=1722064300; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MK9O8/4a3BabvdoGe2yFp5MoF/EC/SVzWLogYQWhbR8=; b=d8/wDWU5l85uko87ANqcU2qeLzUjkwLvUkMuX7jnX0+SqThKUsnWXLTi bRGXhj4wxJB8/4jQ5JzlwcuqztMN4/7e6K1nvnV5SKLisoFqVfLi7r0jy +5cNzkRNUVgYDBK3Z4tl5YykwBbdMYCcF8BBdp/Aw0hoPbGkj0PTEzuEI RGDM4GNLsQF+rTEDHTc4hSM5Fe5erwP0tSYGYcMbZNkQrHvRVDdGy/yPK 38JN2C/Eca7Rz0Ta91Qau+nZ9am/hdODiz9DEtuGrkIxAEkhTAGcEBLIt kHtgARHXEdwJkG3AcnEYdN4b5NrCTDcwjZNQ4xCQJmZDUFFL9AnIWD0KA A==; X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="454888243" X-IronPort-AV: E=Sophos;i="6.01,236,1684825200"; d="scan'208";a="454888243" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jul 2023 00:11:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10784"; a="721162234" X-IronPort-AV: E=Sophos;i="6.01,236,1684825200"; d="scan'208";a="721162234" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga007.jf.intel.com with ESMTP; 28 Jul 2023 00:11:35 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, yuzhao@google.com, willy@infradead.org, david@redhat.com, ryan.roberts@arm.com, shy828301@gmail.com, hughd@google.com Cc: fengwei.yin@intel.com Subject: [PATCH 3/3] mm: mlock: update mlock_pte_range to handle large folio Date: Fri, 28 Jul 2023 15:09:29 +0800 Message-Id: <20230728070929.2487065-4-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230728070929.2487065-1-fengwei.yin@intel.com> References: <20230728070929.2487065-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D09B6100018 X-Stat-Signature: xkgzke4ayzzf1kwfxy5uym8sxb4zmyh3 X-HE-Tag: 1690528300-570250 X-HE-Meta: U2FsdGVkX1+HzeSmPISgbYE9Rh43jcZVD7e/4ccjmFB5jti5QRZB7m21jdyTjfC3Ue3ElGzUBh+lgFTP6Rj1kSU8OW8flYO3y1dZNrtcZnXOzU7rLl68jId61lY8CI7mba2+qYU9hqXZK3zhcVjW0Wj8YSF8Fg5V52ysdihdVNmU46/ei983mcW1Ch0pbQ3OduNEjXJJx19ofnuv9POyOOxs43cwnTgkzBEPuwnxEPJvL9xIeKGpGz2M7s3D8JcX3NqBeFmuCHJ/PMDOfNzXdleo1UZg2+unYFYn4swX0ZP28TzWwT7dIOt1jrL7OpeSo+Qdux7ffKCmVJxa6+DivWPXq9QszEP3KvETBuVlT6GThJ64ihLSryhzIYbZuDALIZYj3P+iRRibXdblKz6Q/jVRsx0GcAQyzBVWOl5z2RFVpbaYwkTJ5nsbWcdA6kQTGDuvCZr3Iz4WX3L7viVTkYLR7AAWE1tuTB9y0SFHzxY4r9d1+y8WZpyMWq0J50jULNyk5VI799x+f74PE/79HeHGZ9WKLwc1zpTT6n05ycm4voDgK+M8+AFJRtx3tTFGdDHyp1Nf36fafXHoeRDKaznXBgzObOOAEfCMkTihpnWsE6lv9ygwEqpr6FZqlSnmsGbQCiwmyb3qLr2uLHsarnpPHOnqcvyBrFuAsCdw+itAQgVeUzEOYtYRfLazYGWVSAGqoRNKu83hdVBTEug9K6pCkPziMoDYsa4UBtpDt3A/HOPeamj9X99ZUt6lQZPV3nZO0TJ1no9dWtJ56u9b8ntdfUG/F4CXnd2VDSkNkWzWAQ6S1CPGKWn7Fcy7Vwsa3fXR8RjzylarH0kKcpOkzewtGCb4m9GYck4HDrKNGTzH7KmB4j7pqz8jKDK7VIFv3P4V6kY4fLWElDiuBl/RlSuKhZ89znvbFwvtjrNKqPiXEUTKXYvp6JlJUVqEcoB+Y9WNkFvOlF3fHCybBnO 9pEY8y58 JGN94pCZ7iaO5/NSlBRbffPnyxeOw+o6XzoCawHZ+HOpWBwDlMG5O+geP+cDqjkTqaKAe6G99lUm6+y1SXBEGARHglhb+2A+B5ebaAs4Cj00kLX1ISUkdgNBuFDUs3IqgohynV+DJ37Wu7VFqd4SvA8UutoGBmnYzt9Xd5VAfumtUofDo0nngUul8nYTSgt9QVzqy0bvdy0pgNvonEAZMUNdR8SzDZ89HFULD/d83xMtxbccWuoR0mfxSnA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Current kernel only lock base size folio during mlock syscall. Add large folio support with following rules: - Only mlock large folio when it's in VM_LOCKED VMA range and fully mapped to page table. fully mapped folio is required to handle the case that mremap happens in the middle of large folio and split pages of large folio to two different VMA. - munlock will apply to the large folio which is in VMA range or cross the VMA boundary. This is required to handle the case that the large folio is mlocked, later the VMA is split in the middle of large folio. Signed-off-by: Yin Fengwei --- mm/mlock.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 55 insertions(+), 2 deletions(-) diff --git a/mm/mlock.c b/mm/mlock.c index 0a0c996c5c21..8056f9176b70 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -305,6 +305,50 @@ void munlock_folio(struct folio *folio) local_unlock(&mlock_fbatch.lock); } +static inline unsigned int folio_mlock_step(struct folio *folio, + pte_t *pte, unsigned long addr, unsigned long end) +{ + unsigned int count, i, nr = folio_nr_pages(folio); + unsigned long pfn = folio_pfn(folio); + pte_t ptent = ptep_get(pte); + + if (!folio_test_large(folio)) + return 1; + + count = pfn + nr - pte_pfn(ptent); + count = min_t(unsigned int, count, (end - addr) >> PAGE_SHIFT); + + if (!pte) + return count; + + for (i = 0; i < count; i++, pte++) { + pte_t entry = ptep_get(pte); + + if (pte_none(entry) || !pte_present(entry)) + break; + if (pte_pfn(entry) - pfn >= nr) + break; + } + + return i; +} + +static inline bool should_mlock_folio(struct folio *folio, + struct vm_area_struct *vma, pte_t *pte) +{ + /* + * For unlock, allow munlock large folio which is partially + * mapped to VMA. As it's possible that large folio is + * mlocked and VMA is split later. + * + * During memory pressure, such kind of large folio can + * be split. And the pages are not in VM_LOCKed VMA + * can be reclaimed. + */ + return !(vma->vm_flags & VM_LOCKED) || + folio_within_vma(folio, vma, pte); +} + static int mlock_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -314,6 +358,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, pte_t *start_pte, *pte; pte_t ptent; struct folio *folio; + unsigned int step = 1; ptl = pmd_trans_huge_lock(pmd, vma); if (ptl) { @@ -334,6 +379,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, walk->action = ACTION_AGAIN; return 0; } + for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { ptent = ptep_get(pte); if (!pte_present(ptent)) @@ -341,12 +387,19 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, folio = vm_normal_folio(vma, addr, ptent); if (!folio || folio_is_zone_device(folio)) continue; - if (folio_test_large(folio)) - continue; + + step = folio_mlock_step(folio, pte, addr, end); + if (!should_mlock_folio(folio, vma, pte)) + goto next_entry; + if (vma->vm_flags & VM_LOCKED) mlock_folio(folio); else munlock_folio(folio); + +next_entry: + pte += step - 1; + addr += (step - 1) << PAGE_SHIFT; } pte_unmap(start_pte); out: