From patchwork Fri Jul 21 09:40:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13321707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52BA7C001DE for ; Fri, 21 Jul 2023 09:41:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D0A772801AB; Fri, 21 Jul 2023 05:41:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB8AB2801A0; Fri, 21 Jul 2023 05:41:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0AE32801AB; Fri, 21 Jul 2023 05:41:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9EB5D2801A0 for ; Fri, 21 Jul 2023 05:41:38 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6F0A012015B for ; Fri, 21 Jul 2023 09:41:38 +0000 (UTC) X-FDA: 81035126676.20.CE84B4F Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf05.hostedemail.com (Postfix) with ESMTP id 5310F100014 for ; Fri, 21 Jul 2023 09:41:35 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=kdwB5dTN; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf05.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689932496; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e2sNh8vTkR820lnb3yYf7b8aM21w60Mfy7WKVZcrew4=; b=4R8P7UJCAoEu5ZmwuDBZJBduhTWTdYgZPM1o27emQlCEd+/d0df7FnY5KAFvRwWBfEo/se RuQlGgreVx7LINU2vn8aOVH18ULp8Mv08vUWEqRrmRxgrOF7z0DP7r66CtT74XuRyizDWI XasxDq9MB7vH4lutsPF6PkS9M52HiNA= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=kdwB5dTN; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf05.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689932496; a=rsa-sha256; cv=none; b=K/Pv1qUKLNSuaJ5bSbUgHeg2Zx7USS3EvcxSiChUMeVr9GUbu3b7FxPOWf+lHaa1a5z7I5 bpWckpDxmGina0n8zsPmM5Bl7hUxB7wU7tEj4NCFMzYREy5KMcdFQDIl7jHak0PL1YCDCV YuQK6xkfeDtnrEpswlbL8ReuIjok9SA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689932496; x=1721468496; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=AU+Bax7o8AGRnodMI5F/Lc+z+q2VSKWl4dvqmzGnJLE=; b=kdwB5dTNHWaWbMK3AuaWmHZ5CG2phibK/ZDDd2vS0FkDkkwAvK70Q8Ci reaPIdAQsXTNuV0mXqxx/2QEK6N66w7K9ml5X521DmbZJU+CEDTwdbQvN nWk1A5byJsVxdkpmm7b40XDPTsgpeA1Fr9pNJ0zU1Ha1g54wtiUOooOrv eg0BOmfFcYRom/cFfLUvkzPvK96REA5bHetwoo0MlHKNy4i0zjkswZTza xwcwtb65irsE6GeJrJgNEVS3elmnIg7d89WSTx9jOHx2bz1S43ooxi9D1 AURjv3t8MIAPG0toQimJH/w5VwX3QH4F1GrxtGNpmoDeEM/i5G78UgPFw w==; X-IronPort-AV: E=McAfee;i="6600,9927,10777"; a="397874317" X-IronPort-AV: E=Sophos;i="6.01,220,1684825200"; d="scan'208";a="397874317" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Jul 2023 02:40:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10777"; a="898661109" X-IronPort-AV: E=Sophos;i="6.01,220,1684825200"; d="scan'208";a="898661109" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga005.jf.intel.com with ESMTP; 21 Jul 2023 02:40:46 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, minchan@kernel.org, yuzhao@google.com, willy@infradead.org, david@redhat.com, ryan.roberts@arm.com, shy828301@gmail.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH v2 1/4] madvise: not use mapcount() against large folio for sharing check Date: Fri, 21 Jul 2023 17:40:40 +0800 Message-Id: <20230721094043.2506691-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230721094043.2506691-1-fengwei.yin@intel.com> References: <20230721094043.2506691-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5310F100014 X-Stat-Signature: kbh4h7f4jzmmsw64omwfwpcckoqhcbno X-HE-Tag: 1689932495-451696 X-HE-Meta: U2FsdGVkX19LlUIypKzALV3194CxeXvx77CLTQ4vUs1Nb6DMdAMGzsZjyF/vlyMblrFzozRQVzkyKgKybIUFKJe+LaiQ5Ss/ZIPrtwg4N1yryF5qOsh32tj5tr5cpUxp3dTqUq/0FoFz0nAi64lq5wYUMj3ErWqYH9UAbXkqT3uh8bTslgs1Gk1MehvZasJDcy4u2WhAxPUZXh3zZVcE7Y85m+x0R5pmhd6yUiT7Zkz2Td9+YH6tcwoAbc+gRgvPipvwvy48M0fXb4e9OysKGyJs7EKnTuIDueN8EKle9sNt9XbFGx3+5iLJoIqGNF8uQ1PNAyjTsk+rIDScevAGZCDNqJ99d4S4kOF6hNqH7MBz8Y7xGQ+fLOeYqWgXZV1QFErPTMbO2W8jLjaL8ylIDPRnXLqyhA1HPiqd6BP1RMmymBXwIXy1bBC7bYCi9OwB0rjK8wimb9wF/Eyt09J8o08vrcmFzmIYOyJmEyCFuo4lPXcdGeBODl9USy8Towa98RSfhzOydFOspKgtaaa6ekWAKnZZkJ2e1GtLXogDyyQRGA+son6D2/wIoX5R8xq1OzTkDeK6uvf815e4nZob4/eyblOOy9xW1hq99fL1V/6FH1tSxp129p6I68JNVHiAWqs2dI1iWGtp2L0hjRsU5vqwa/iTeFSZPg9V/36awHcYwQ8wck16qK/yv2huxWqu4K4m9jzVCmmPDG3nWjMrXzzjgD0W7CPv/6g1iIMSD77FRKuYDTM05ILmL3FpvwKLt30OvnPbLuwGpUefy2bctUaIU7g3ZCqqlsFBvtdfRRMNSAKLbn9RMKehBTNxalOAp5eX57FZ1ippMNSCFaZldKwxycUMAHmnfcHoAwZqT9RdoqvtDqyKLMT2t4ZrypYfpSi4VRCAkD0LT24Dcrk5lvu+kUpPSB8mTKDb2o61HFE5HmOBVd6uP0xyzCA89wGuzqlg7Dv/C5gz33ofpKR Im5Kf6Mt /gAVcFadc98RxSnpinp4WL6iWfTHq9bXT49Bi/NoSp2JyYyNhCHIaBrIo0AwtuRvWp3JdfF6SmFZW3a1VEnogQctTadXGKvRBPK4Eljxvf+Zha1aFCeohTnFO6r+Zc7yAjAYMSVg+hcgItKZfuIs4vQ73Iw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The commit 07e8c82b5eff ("madvise: convert madvise_cold_or_pageout_pte_range() to use folios") replaced the page_mapcount() with folio_mapcount() to check whether the folio is shared by other mapping. But it's not correct for large folio. folio_mapcount() returns the total mapcount of large folio which is not suitable to detect whether the folio is shared. Use folio_estimated_sharers() which returns a estimated number of shares. That means it's not 100% correct. But it should be OK for madvise case here. Signed-off-by: Yin Fengwei Reviewed-by: Yu Zhao --- mm/madvise.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 38382a5d1e39..f12933ebcc24 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -383,7 +383,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, folio = pfn_folio(pmd_pfn(orig_pmd)); /* Do not interfere with other mappings of this folio */ - if (folio_mapcount(folio) != 1) + if (folio_estimated_sharers(folio) != 1) goto huge_unlock; if (pageout_anon_only_filter && !folio_test_anon(folio)) @@ -459,7 +459,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, if (folio_test_large(folio)) { int err; - if (folio_mapcount(folio) != 1) + if (folio_estimated_sharers(folio) != 1) break; if (pageout_anon_only_filter && !folio_test_anon(folio)) break; @@ -682,7 +682,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, if (folio_test_large(folio)) { int err; - if (folio_mapcount(folio) != 1) + if (folio_estimated_sharers(folio) != 1) break; if (!folio_trylock(folio)) break; From patchwork Fri Jul 21 09:40:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13321708 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00372C0015E for ; Fri, 21 Jul 2023 09:41:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0ACE02801AC; Fri, 21 Jul 2023 05:41:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 05C952801A0; Fri, 21 Jul 2023 05:41:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E400C2801AC; Fri, 21 Jul 2023 05:41:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D3EF72801A0 for ; Fri, 21 Jul 2023 05:41:39 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8EACBB114A for ; Fri, 21 Jul 2023 09:41:39 +0000 (UTC) X-FDA: 81035126718.10.63B7311 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf11.hostedemail.com (Postfix) with ESMTP id 4CBC04000E for ; Fri, 21 Jul 2023 09:41:37 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=IQ4OQMpU; spf=pass (imf11.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689932497; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=apzMRyTxT/YpU0tZgW/dawYoZpV0o4D5ZoL/b9Sz3J8=; b=kVLQcaWIGtmeESl+O09Kh04K1mj35oSedESotg6LETez36l0UQxTofSLNH4y6terwBepP1 rtVC+duH6rhTE4bQHsXz7v+PmzAT79BKsw6ZWmmL2pbuTwQBiw/oxtnDMrLkLmWIRg8SWZ RFVPiP5D077Gaf977d0VqiwE4M+m1NE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689932497; a=rsa-sha256; cv=none; b=iZGRgPmql9Q9a7W6e6et8poaBwfwUFZOOWc0MaShC1C0hm2QJkgbzQUmcvu/WQsy42iFiI 1A55CFla8uHiwPNYSYyHfxw9bajC2COBHcGbAsIyC2vP1iKlZjHPS0Cbadhw6vJ0IZHZg4 s9J01MERdfTQ03zfYxpEnMVBgBrBQ5I= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=IQ4OQMpU; spf=pass (imf11.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689932497; x=1721468497; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cu8lT3oOEGsTcv6kJqkYTpbtDaklUJpbsDBtNyUKlKw=; b=IQ4OQMpU3blp7bbrJDxW1aQqXGbor3hoIhbK7cYiyFOCkOeBvyXM4CpQ XP3Hx4d7orp6XtlGA1lubFcmZgwNedW0QSDcq2eK/LE7f2qQBUmpcdCEh rY2bU1Ib0wgFSZemKM3Hy8GSOcgtrqklbIdCJPUQ2IGlKPi1kksKjje1o p8TTQBZ4zGRbPpcZVgcdsXeCWPoCUQj2ecwDHbiiKWxwYHDOfKHETHGmi YC9C8SDUy8dh4+N8q8+IyJUysyIyQR1tH3U9NuHnaF1OewtFP6KzeQTA0 66nSWeIqW9GAs8LVMviFOMmFG64xafxHVi323yJBMIW3qb5bbozCFTjDa g==; X-IronPort-AV: E=McAfee;i="6600,9927,10777"; a="397874365" X-IronPort-AV: E=Sophos;i="6.01,220,1684825200"; d="scan'208";a="397874365" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Jul 2023 02:41:02 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10777"; a="971386942" X-IronPort-AV: E=Sophos;i="6.01,220,1684825200"; d="scan'208";a="971386942" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga006.fm.intel.com with ESMTP; 21 Jul 2023 02:40:59 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, minchan@kernel.org, yuzhao@google.com, willy@infradead.org, david@redhat.com, ryan.roberts@arm.com, shy828301@gmail.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH v2 2/4] madvise: Use notify-able API to clear and flush page table entries Date: Fri, 21 Jul 2023 17:40:41 +0800 Message-Id: <20230721094043.2506691-3-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230721094043.2506691-1-fengwei.yin@intel.com> References: <20230721094043.2506691-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 4CBC04000E X-Rspam-User: X-Stat-Signature: nwtkf1xt389mn3cs9dum41kw9ox8mnpi X-Rspamd-Server: rspam03 X-HE-Tag: 1689932497-272248 X-HE-Meta: U2FsdGVkX18J95nUbX6fgt9HzVrR/uRknA1Gluch6xeFkSZK04hFfBVgEefx6imdrxFrZi4PQMgjAeW8BFfIZ1sRJXekXS0FN8n8Y0RYQPYqG+78kuiym0/zjTpTheJkMgSlxPYEGMyGl2wo7Qi7WnbMcMDwPqsnso+ZP885kLwJakvnbPUGW4pwCMYJvedlSz/EbKJRX/cnPJIqof8QtRZd4QkfjGoTGZllyD4XfNYOpBQ85YGmNYZXsO7KE6nyCDBLhP6S3PD837eNJj1k2EQXx5Y/FnU4GVeHWmxCMMwIsWkMXKgECoprr9U7Zk8h3ycHv8ur33PjyG0+TfQOvdpasmQcRnLLdkmZmuQr4iUSVqqLCTyeWZ8ty727tgOoKW7zEjoNk3X4kKcwOd3+nkucobTBc4mU4IMEq+1CXLCZno9L/qjCuT6FQrnarz9MQZZ3nVg9Njh1VDuGzgul0Dr54SmTXHi81KkUqP4EBhZ5gUNst+il/rjPl/U+GXcCkf+2UuRjRrXZ0LKU5jsb1W8tFdHTse7PJXO5fJmwEdB/x8arDAh4Y831KWDLP1Vw+Pt54GTWxN2WW3+TGZ0R8itfOtme//w3EuG7ir3HW41xohivbUtJMWkbUNGhhWF1ruXGS7NGSL2W6EaAorrQslLx89JBVPeUgIGgxF0xJcSKYxkmUBumHrJO+N71o6qGvt4H+UMHzyBgwJWbkqjPorHFqrWjVL++oMW8mSi8cThpecKnSkGpc6XIlPTXHXPKf4eDcTZwQxuHfnIkbo2C0wlslCffq7M4UYANbE/30jnAswGsZMilB8J+jllSVFDl6N7pDZPEWDfkvxQ8lOiFgd1+FgRY5H2a0PGo01zIRIPGF06la8QacNZmR51jmh6P6n5UKE8Wn39S+LupMj65CET9GD0rxc51XMyyfpJQRp8hXOxjNCFQx3rlIPN9q74biveJaV8WypG9kw6hRaA OVK75ktI 61uLP7hVShzk9vyJTJs5QEqp15BEQtAvcPWJ1gH8Hjbg7l5DGi8eUy9hsIdIrhTNR+ZhuEBag0Z/QEZjLT8rPgKE4zJgv8YVYPKioBmTZb0KrSW8BSTkdONrUZrTiuK0W2a7WlS38LHl3qFsXwUzyYBB6BQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently, in function madvise_cold_or_pageout_pte_range(), the young bit of pte/pmd is cleared notify subscripter. Using notify-able API to make sure the subscripter is signaled about the young bit clearing. Signed-off-by: Yin Fengwei --- mm/madvise.c | 18 ++---------------- 1 file changed, 2 insertions(+), 16 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index f12933ebcc24..b236e201a738 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -403,14 +403,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, return 0; } - if (pmd_young(orig_pmd)) { - pmdp_invalidate(vma, addr, pmd); - orig_pmd = pmd_mkold(orig_pmd); - - set_pmd_at(mm, addr, pmd, orig_pmd); - tlb_remove_pmd_tlb_entry(tlb, pmd, addr); - } - + pmdp_clear_flush_young_notify(vma, addr, pmd); folio_clear_referenced(folio); folio_test_clear_young(folio); if (folio_test_active(folio)) @@ -496,14 +489,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - if (pte_young(ptent)) { - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - ptent = pte_mkold(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); - } - + ptep_clear_flush_young_notify(vma, addr, pte); /* * We are deactivating a folio for accelerating reclaiming. * VM couldn't reclaim the folio unless we clear PG_young. From patchwork Fri Jul 21 09:40:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13321718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF54EC001DE for ; Fri, 21 Jul 2023 09:41:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 53FD72801AD; Fri, 21 Jul 2023 05:41:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4EF112801A0; Fri, 21 Jul 2023 05:41:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B7392801AD; Fri, 21 Jul 2023 05:41:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2BCDC2801A0 for ; Fri, 21 Jul 2023 05:41:57 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 00F8540209 for ; Fri, 21 Jul 2023 09:41:56 +0000 (UTC) X-FDA: 81035127474.15.AE9C532 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf09.hostedemail.com (Postfix) with ESMTP id C2B81140015 for ; Fri, 21 Jul 2023 09:41:54 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="n8InHVh/"; spf=pass (imf09.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689932515; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CJU6rlhjEkJQvGttexfzxf5+8AjYlAy+5TBBCbgKccE=; b=dJKyQTQ4KoPbJxgZeJgLi3D2dtVHMryyDOIkLdhRphl/mLccRRVbJigRu1qFflmgxjVR7H pULOgGTxEMw596FbaPE+uA/6lKKv2DXExe8z6ZeW4BmO/1YYSaawxG7TDZRF8NfVomKcaE kGn5tXSZWH9xhmceMob04qTYMJa8W2A= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="n8InHVh/"; spf=pass (imf09.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689932515; a=rsa-sha256; cv=none; b=fFV/LEFeLrmZmTL+g2s87ai1ZRpfLspz7DeZqdd36e8MWhIAGljaoWyUm1/tcEuj5Vt45r 59i6GrwfO6Ub2qQfhELP2j5i5CilLaHkkru/JBZ2ifMYdK9kQLQfwBLhIzu6tazRVzMsru tkfCJbr/2RPwOJtPl5Ohmxpk61TbdL4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689932514; x=1721468514; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gTxpEd4Cqo7yQGmFIHWT43Boa+nqgiZNKOPooxf76ME=; b=n8InHVh/he2wzC/Q27oveMji9JDAK8n62B6j0FoBMdVlhpipK2YNYQ8H v9XIAs1nw9i1o0YFgBw+pDQC8QAwDezHE7CZkx3VmYvgEGaWUURWb/h4O 7XDN23l4hdiqUG2sRWpTaozJP4QUffFneGSOGhzGiYOquEyHyFngtWTNl GJIlKcK399nlcVMxhKh6R2tlfpGlOhkWjNb4ryWsKNhLXScrX4TSWaEnS ekS8lNI7Q/kZPPWyk+0OBveGPxCISGhDLypCUEZt/4183VSHFW29fpwTs Ytib2V5LD7rkqrWfmUASgnrIJOXUVYolFiLWoec7WcxEE44dMPO49RyCE g==; X-IronPort-AV: E=McAfee;i="6600,9927,10777"; a="346575456" X-IronPort-AV: E=Sophos;i="6.01,220,1684825200"; d="scan'208";a="346575456" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Jul 2023 02:41:15 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10777"; a="838480273" X-IronPort-AV: E=Sophos;i="6.01,220,1684825200"; d="scan'208";a="838480273" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga002.fm.intel.com with ESMTP; 21 Jul 2023 02:41:13 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, minchan@kernel.org, yuzhao@google.com, willy@infradead.org, david@redhat.com, ryan.roberts@arm.com, shy828301@gmail.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH v2 3/4] mm: add functions folio_in_range() and folio_within_vma() Date: Fri, 21 Jul 2023 17:40:42 +0800 Message-Id: <20230721094043.2506691-4-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230721094043.2506691-1-fengwei.yin@intel.com> References: <20230721094043.2506691-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C2B81140015 X-Rspam-User: X-Stat-Signature: nggh7o7mftu1rpo4bs8s9933qtkwtje9 X-Rspamd-Server: rspam01 X-HE-Tag: 1689932514-572173 X-HE-Meta: U2FsdGVkX1/bta261Ye5PB0ee/BuDPit4b/oOojlkHLIokPQM1pb9HxU1cqJLByluZkII1ax6VGP8uttjj1R22S9184pbL4T0VSvTBuY9LwsttG+VB8n5s6WkrkoGLLf3R08tzc7wcyD/8PuM3clshgHMlacEIwDjyebXaYplUViTgusMUImRjO03hTAUZdnTSx0NWzkCdbjqHX/AAgQ5/u+T0/Xed3PtaL5UEMvf9mPrEWKbx5MneGN5loWlbwgv/O0lsy+04ZMi+zdLtPB7o77aGbCTS28uRunOxbi4WWUCprsyOpnzPNkPjWQ0UQuejtooaUHbqjcQcym6q2qQZbzE/iAb2fLcxGdWn6EsEj8qGSPbV+tViGssEUV1hFJu0UkFcgZKU3j1vlFaT7bvjQzkpEvoGOiXkLvNk1KznwSS1UNWGtHpxnMAe+nxPdghrURVNC++wxyvAtBxs21TipyzAy79DrNXUnEek/A0ZLfKZnSmKa96atBkR+m8zOybEZJrED9JEAUDTVetvqy9VeA30BNWgBzmAvobSVORe4hTZZ/wzyOsgu5yY6h/eLdn/wuDiBEOyksVF2K05S3gwmSFiMQC+Z35r/jqN6Gw5liXu4ZNNWXwCOme+WhwSNYrRI1bnxAcJ1tVfq5MrC2KdwiSAbuh8YqhasCVneNh8rU5JPA4HQIfViqZXbcOxVtPuWzuPoEQxnZb/H98knXiFdlzKDOgT7sf4a0a+FmVi9uXjI4nmfRa03wNka9XOo/z9QxPpH9TX70fkfbYxgd+Ll4TNq34iyTk0dNZHUotnLZ5topb3PDGuKd/Towo1MCnJVeE7CxLUnALK1K1vZl+SY83rX4QN+3pgLb1jMvT97MCXGTB0yTwZajox4eWr7m0AzxiL0ghmrBoIX3jDpqCRUYprz/SN5SbshjbxodJASPmKnhTxfaCHragG/gxJerOqdoLNCDkHAff/096Q6 qZUBtate BhXNfoza5CCZpdg6CUB6WdWcpy1MXRHH/PW1JojnojkgIyV+etvx6cw8xA0X9fVfVM43NfjXLXSGx7GGhJFBlEzyfDiCUw7w/gIJUgpEpLiOj8OSVnEc0q64jPwpzy/CB2yfZ4Lg0kYxJ2Lj1p3s2iGCMJ7n0W4T///AoM5IkuFBgBpMryBcOWrFj6xyNMV3G/sq0cjvV7Ow93PE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It will be used to check whether the folio is mapped to specific VMA and whether the mapping address of folio is in the range. Also a helper function folio_within_vma() to check whether folio is in the range of vma based on folio_in_range(). Signed-off-by: Yin Fengwei --- mm/internal.h | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/mm/internal.h b/mm/internal.h index 483add0bfb28..c7dd15d8de3e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -585,6 +585,38 @@ extern long faultin_vma_page_range(struct vm_area_struct *vma, bool write, int *locked); extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags, unsigned long bytes); + +static inline bool +folio_in_range(struct folio *folio, struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + pgoff_t pgoff, addr; + unsigned long vma_pglen = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; + + VM_WARN_ON_FOLIO(folio_test_ksm(folio), folio); + if (start < vma->vm_start) + start = vma->vm_start; + + if (end > vma->vm_end) + end = vma->vm_end; + + pgoff = folio_pgoff(folio); + + /* if folio start address is not in vma range */ + if (pgoff < vma->vm_pgoff || pgoff > vma->vm_pgoff + vma_pglen) + return false; + + addr = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); + + return ((addr >= start) && (addr + folio_size(folio) <= end)); +} + +static inline bool +folio_within_vma(struct folio *folio, struct vm_area_struct *vma) +{ + return folio_in_range(folio, vma, vma->vm_start, vma->vm_end); +} + /* * mlock_vma_folio() and munlock_vma_folio(): * should be called with vma's mmap_lock held for read or write, From patchwork Fri Jul 21 09:40:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13321719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 240D0EB64DC for ; Fri, 21 Jul 2023 09:42:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B32932801AE; Fri, 21 Jul 2023 05:41:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE3982801A0; Fri, 21 Jul 2023 05:41:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 983F42801AE; Fri, 21 Jul 2023 05:41:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 860FB2801A0 for ; Fri, 21 Jul 2023 05:41:59 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5AE27C01EC for ; Fri, 21 Jul 2023 09:41:59 +0000 (UTC) X-FDA: 81035127558.22.9AE5CAC Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf09.hostedemail.com (Postfix) with ESMTP id 3DBB8140006 for ; Fri, 21 Jul 2023 09:41:57 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=adCbNm7l; spf=pass (imf09.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689932517; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XHg8XCtLoVnx+178so0LHeAi7YesWlPOQ5pb0at6+2Q=; b=XW5T3H0C32NzbZegbaykz9YlG4YAZvunYXlpDZt1fimzIIYNh15bODI+EN7ZRGFwS95mVA 29EhtvaeCLt9e5oeEpBuwpZcm53KZ1UgNHslNBB6gan1XUEt2j3iOTOxQpeYq11FBjAf2Q UN8DQzHwD1dzaPr9qB9H4vP/T+GqwP8= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=adCbNm7l; spf=pass (imf09.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689932517; a=rsa-sha256; cv=none; b=Iu+GvvRuz2soE2jzaUeURVj2uTA7tHo+7VsRk9m2VewRQtuMyTafMTMWWUo9GH0wbxmmo4 tYDz0uGQsJVLYF6Y6JS2bEjhT2zrqRqA8aevHVQIo4/OiOolMB+DqYfiywcmBLmY3DRNXc A7kDOctqcvIrg1hB2VbIjEuVDWA7KL4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689932517; x=1721468517; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Y7g87XRsxPlXPl3gcejHKFouz+BtPVkulKmHSjYpyg4=; b=adCbNm7lQZ63xMlikNu4u9BSZiNj6FUfQq+uwnH1h0NKJ/C2ca/2L0Qg 8whdPE6ipiBYCYwzsQA2ttKvOMrw3YMqgbA+YM856TDZ1R2bqmYKZxfiJ Cjc19p1m5W4DWkCEsGVuwOHEgqxKrHezIY91TyQI6Xn770zDVfXxhegGw FqlsOZMejH9sk3Q85XnTBnD/72PX5Xx9aD1qrbmxDT+otnq7EFXvqeQfb d2OeFCljOzcs2AFt/z3Lrfj6iamfCI4PDpOyKGu9oF3JW3Pi3RquWYuNX d+LyA5ZmlVze3Sklo41Vj773xmPSLZQe6TeuPdmnzv9zYZ3RJPZH7p4yV A==; X-IronPort-AV: E=McAfee;i="6600,9927,10777"; a="346575498" X-IronPort-AV: E=Sophos;i="6.01,220,1684825200"; d="scan'208";a="346575498" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 21 Jul 2023 02:41:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10777"; a="838480297" X-IronPort-AV: E=Sophos;i="6.01,220,1684825200"; d="scan'208";a="838480297" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga002.fm.intel.com with ESMTP; 21 Jul 2023 02:41:26 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, minchan@kernel.org, yuzhao@google.com, willy@infradead.org, david@redhat.com, ryan.roberts@arm.com, shy828301@gmail.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH v2 4/4] madvise: avoid trying to split large folio always in cold_pageout Date: Fri, 21 Jul 2023 17:40:43 +0800 Message-Id: <20230721094043.2506691-5-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230721094043.2506691-1-fengwei.yin@intel.com> References: <20230721094043.2506691-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3DBB8140006 X-Rspam-User: X-Stat-Signature: tu7rnd3jqxszi7u76dya5exdh3g8rfsd X-Rspamd-Server: rspam01 X-HE-Tag: 1689932517-70164 X-HE-Meta: U2FsdGVkX1++ueX6Ia86VABXGI6qAwNJDrL4WWZb7t7btUq01jaIMzoQ6FWfYkA8/qy7kxq+QBrnpnGP5LaWzYvXizlYUsnFwlQ83yyZirIxPEssUYULPZqPFTL9dlGxI2oqsQ27HWI2jPRTOQW2yvLTWNGhxEFQF2ZgEZfi97/69OGqsf8Yk0K+Abk+ccNPgfAjHcfh7C5oHf2oCn1ClDagRZ7y6Yj3ZZ8N8tVQiBKMxL+K2HXfh+OVrourjezZnsK74gAGYn5FK8E55HbnqHBGX7CY8ZmZ19rHrMS1LxEf2VnoHmK3KfmHLhqglE0DirPaQ3sM/93NqFy47sf8BQ45S3xftMSFibCJxEzm2KTRkJhRHqKOAdtvY9FWPi19xIhabXYeysgnpQWCbbx/+LU5E9Nfk0pDZw95XgbLxijSZUDxCm0A7obzRwK4OnODH0CBqLhn0FTuQDnTYCi9a9VZ5ZcajHbY6EpGjUUokLoOmPV7R9/NFK+/JHOVpscTYm6ITjK0ffa1hMRP3xdVjV1IAVTJet9Rd6ipuT2hl2k/LmOeq/S1IhWKXyUia+quyXfx93n51OemACtXPfTIHRDPnm2nB8y7Jz6k7ANAGQGwMDB0r+8SdigWgOMTvwKFJArdz6blXDg1aFYQEpEnOW3eylJLb0kPblLUfEv9wjIHwySJLYOrjfr7qyVnByaEhPs5QEq3VGP8ccAP0oDojYoKaYlOuYLO3aa6Ui801kw8uhpCszzrHp+hZSfeGRwtZOd1lsF4m0yTsK4+HSIOqkV86UHvI8LYI/vBxKagx2UZ3mc+sUh9sZX5QUmVD54mlDeyyt9nJFE8p7+Ye+hEpXlSDECV8c3uE7nkGUzT6JBKT7DGr2z4nRioV16Njc1cdWAjdizUQNVW1zmgJwJCkmSAukfRpTJ9gBJ5cRvTZPLoAAdOHrnTvLX+1vVYNI86+PMm2VgI7abuF4dP2he 9oCokNZQ I1b4En9eOZhGcyrvfiXCv3FGfTjanBBYqeoJOENfmg2UMNoq97OsXPAehLrC6skVKMg4/Omtm6+0hE89jyWeLmn0Iz6iYT7p1qTb7MvXN9EbMPYRPQp9l2njhMyMoms8eLJx4vzGV/pU482i59DcjARwF5CKd8KAM1pwgLyXCKSSS9209ZLls/ajkQMV45ou51dz2q4un0N82QsE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Current madvise_cold_or_pageout_pte_range() always tries to split large folio. Avoid trying to split large folio always by: - if large folio is in the request range, don't split it. Leave to page reclaim to decide whether the large folio needs be split. - if large folio crosses boundaries of request range, skip it if it's page cache. Try to split it if it's anonymous large folio. If failed to split it, just skip it. Invoke folio_referenced() to clear the A bit for large folio. As it will acquire pte lock, just do it after release pte lock. Signed-off-by: Yin Fengwei --- mm/internal.h | 10 +++++ mm/madvise.c | 118 +++++++++++++++++++++++++++++++++++--------------- 2 files changed, 93 insertions(+), 35 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index c7dd15d8de3e..cd1ff348d690 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -586,6 +586,16 @@ extern long faultin_vma_page_range(struct vm_area_struct *vma, extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags, unsigned long bytes); +static inline unsigned int +folio_op_size(struct folio *folio, pte_t pte, + unsigned long addr, unsigned long end) +{ + unsigned int nr; + + nr = folio_pfn(folio) + folio_nr_pages(folio) - pte_pfn(pte); + return min_t(unsigned int, nr, (end - addr) >> PAGE_SHIFT); +} + static inline bool folio_in_range(struct folio *folio, struct vm_area_struct *vma, unsigned long start, unsigned long end) diff --git a/mm/madvise.c b/mm/madvise.c index b236e201a738..71af370c3251 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -339,6 +339,23 @@ static inline bool can_do_file_pageout(struct vm_area_struct *vma) file_permission(vma->vm_file, MAY_WRITE) == 0; } +static inline bool skip_cur_entry(struct folio *folio, bool pageout_anon_only) +{ + if (!folio) + return true; + + if (folio_is_zone_device(folio)) + return true; + + if (!folio_test_lru(folio)) + return true; + + if (pageout_anon_only && !folio_test_anon(folio)) + return true; + + return false; +} + static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -352,7 +369,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, spinlock_t *ptl; struct folio *folio = NULL; LIST_HEAD(folio_list); + LIST_HEAD(reclaim_list); bool pageout_anon_only_filter; + unsigned long start = addr; if (fatal_signal_pending(current)) return -EINTR; @@ -442,54 +461,90 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, continue; folio = vm_normal_folio(vma, addr, ptent); - if (!folio || folio_is_zone_device(folio)) + if (skip_cur_entry(folio, pageout_anon_only_filter)) continue; /* - * Creating a THP page is expensive so split it only if we - * are sure it's worth. Split it if we are only owner. + * Split large folio only if it's anonymous, cross the + * boundaries of request range and we are likely the + * only onwer. */ if (folio_test_large(folio)) { - int err; + int err, step; if (folio_estimated_sharers(folio) != 1) - break; - if (pageout_anon_only_filter && !folio_test_anon(folio)) - break; - if (!folio_trylock(folio)) - break; + continue; + if (folio_in_range(folio, vma, start, end)) + goto pageout_cold_folio; + if (!folio_test_anon(folio) || !folio_trylock(folio)) + continue; + folio_get(folio); + step = folio_op_size(folio, ptent, addr, end); arch_leave_lazy_mmu_mode(); pte_unmap_unlock(start_pte, ptl); start_pte = NULL; err = split_folio(folio); folio_unlock(folio); folio_put(folio); - if (err) - break; + start_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); if (!start_pte) break; arch_enter_lazy_mmu_mode(); - pte--; - addr -= PAGE_SIZE; - continue; - } - /* - * Do not interfere with other mappings of this folio and - * non-LRU folio. - */ - if (!folio_test_lru(folio) || folio_mapcount(folio) != 1) + /* split success. retry the same entry */ + if (!err) + step = 0; + + /* + * Split fails, jump over the whole folio to avoid + * grabbing same folio but fails to split it again + * and again. + */ + pte += step - 1; + addr += (step - 1) << PAGE_SHIFT; continue; + } - if (pageout_anon_only_filter && !folio_test_anon(folio)) + /* Do not interfere with other mappings of this folio */ + if (folio_mapcount(folio) != 1) continue; VM_BUG_ON_FOLIO(folio_test_large(folio), folio); - ptep_clear_flush_young_notify(vma, addr, pte); + +pageout_cold_folio: + if (folio_isolate_lru(folio)) { + if (folio_test_unevictable(folio)) + folio_putback_lru(folio); + else + list_add(&folio->lru, &folio_list); + } + } + + if (start_pte) { + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + } + + while (!list_empty(&folio_list)) { + folio = lru_to_folio(&folio_list); + list_del(&folio->lru); + + if (folio_test_large(folio)) { + int refs; + unsigned long flags; + struct mem_cgroup *memcg = folio_memcg(folio); + + refs = folio_referenced(folio, 0, memcg, &flags); + if ((flags & VM_LOCKED) || (refs == -1)) { + folio_putback_lru(folio); + continue; + } + } + /* * We are deactivating a folio for accelerating reclaiming. * VM couldn't reclaim the folio unless we clear PG_young. @@ -501,22 +556,15 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, if (folio_test_active(folio)) folio_set_workingset(folio); if (pageout) { - if (folio_isolate_lru(folio)) { - if (folio_test_unevictable(folio)) - folio_putback_lru(folio); - else - list_add(&folio->lru, &folio_list); - } - } else - folio_deactivate(folio); + list_add(&folio->lru, &reclaim_list); + } else { + folio_clear_active(folio); + folio_putback_lru(folio); + } } - if (start_pte) { - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(start_pte, ptl); - } if (pageout) - reclaim_pages(&folio_list); + reclaim_pages(&reclaim_list); cond_resched(); return 0;