From patchwork Fri Jul 7 16:52:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yin, Fengwei" X-Patchwork-Id: 13305123 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7215EB64D9 for ; Fri, 7 Jul 2023 16:52:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 612618D0002; Fri, 7 Jul 2023 12:52:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C1EA6B0074; Fri, 7 Jul 2023 12:52:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D7D68D0002; Fri, 7 Jul 2023 12:52:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 407F46B0072 for ; Fri, 7 Jul 2023 12:52:40 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F0BB3140D40 for ; Fri, 7 Jul 2023 16:52:39 +0000 (UTC) X-FDA: 80985409638.15.6C36755 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by imf03.hostedemail.com (Postfix) with ESMTP id 8D4C220016 for ; Fri, 7 Jul 2023 16:52:37 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=achHiaMf; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf03.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688748757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5IdTAm/mVq4hBk7ISMJ8Drz/LzCia5XtyQmyhwvy/ik=; b=tDJWsYuW3CaEjV/dbWWzPuJWLZfiQ3tT1YFH3A/l+De2HW4lyxNzpl5VkN31o4acX6hbQY YMhlpK4PY57gl6b4D1tp4dc+vgaMpm2+AcoULTZJdKUF6O0LNqvQB7AuzEETH7ZiTnPZQ/ pMa5Fi1CFsqS7vsVi9lSuNQT8LQWRZY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=achHiaMf; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf03.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688748757; a=rsa-sha256; cv=none; b=k/u0RxwyAXubIEFHBRN/1XtbFtE+e0zAiSfn4RBLLU8q4NqH8Ucc5JRJZFFPmW00IRO5d2 jPiCX+F504BOO37lumobNkegq5awpZVbrnCmYvKD4RMB+vsl/R+9upFaEEXWf0T75qUuJw s0I4EMurhdzPgVxjXlEjuTQ0M1iX6Og= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688748757; x=1720284757; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9AZoEU1dBDfQPpclAW3AxhTYvcE6HGN/KVZD4GlGeoo=; b=achHiaMfUAiJypiVeDdj8zrGAhPX5u0U19gIPuwqd3o3SlIgghoyHlLX WkVhXCC/NQEH01X2NtlVVvr5ei5zMysjsi9dn2kz3tLH/EWwljdBY+4tr JRsHeO5sriOM2XU0+h+XX2xFcn4PfgSiHnzKlRFCaYZOdqp0IWyUVAr4P P9sIH+J1hkGw0QVAhWvuhCM+vcvpmNuIl6w8nQeFRchO7AizvmW5sPNb5 xp+34tV5XQ3V6j6VLvEJYdxp94+SZbDbG0NShmCEUjIVw8VcGFTwut5aD yrPtQz3I8agblg0XovqGw+JxS9H5EjnXDqI0E4l8M3l2xjoge7IKvFbXt w==; X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="353776317" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="353776317" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2023 09:52:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="697290444" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="697290444" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga006.jf.intel.com with ESMTP; 07 Jul 2023 09:52:33 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, yuzhao@google.com, ryan.roberts@arm.com, shy828301@gmail.com, akpm@linux-foundation.org, willy@infradead.org, david@redhat.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH 1/3] mm: add function folio_in_range() Date: Sat, 8 Jul 2023 00:52:19 +0800 Message-Id: <20230707165221.4076590-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230707165221.4076590-1-fengwei.yin@intel.com> References: <20230707165221.4076590-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: qi3gzyb9bdkbwjj37nh5g4tut7rze58c X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8D4C220016 X-HE-Tag: 1688748757-180509 X-HE-Meta: U2FsdGVkX18UyWOBYAYxjhaHtjqg0mcWbsJ4Q7SUZfkj4OfoB7haXCO6muruN2h2qycJ7LcE3rLQwVY6T3jgFySXGtW3UrfMLadO7Fu8x6kkDF8mKcAKRMqzd1Cjzu+6G+S6ncHg5M/vgOVFTFa6gPn7SnPB/6WLJTmFKY3CFp55D5CagmC2zGky3VQElEKD0cTNThF3F8srU3179LbqjLJoK7sdLSDUHV2ixbZuLXSKnALQjTcnNKjI9pb+9syLXeWpijDUAoDl+m8zO2on6TTyMLD9myerR6XMR4sX3vEdYGTAGLalcvN5QWtYG37Euc7UHUhiwjgKWHo19flyxUPF0dhpYnt0cpaWyIBNFDStmosh0+wr8WfH4OMtDxvf1OatpBBRZ5V78tBCX3rcvq99BjN8BlCYIk6YSD9GlnDfezFg9qNYmYX+lPj5B/+B+8p3rD72zh5OAv33BXZlnlSghX6ihrXiwi3L+mxcCPSydSaC+i8XDOS+SrUSZOHpk8318BsvaePdYAuEg4HggIGzpMXwLo1+yjynqzuV5OA+ycXrESVfdsgZ5F1CmLGKSm6ylGVEfwL9MxTRix1NN/okyUx8+Ak0aeDFq4Y68OL8U20rW0h0EdqYeMrlkdXnNiUqZCsP/v46eR6KfNikPyBsvKu+Qq3M32EqVw1z0395vlpCYyg3bmGiI+Jnl5XYla3cWKK1f1aQ4kDiYMfvP97BtPDu7YVgPC7p/MXcKcyHhIqhHguysyh80kJzYy0P+wrv/8zB9BCOvIQqj9+QT7l7BboqXCsbV7qU7YXjqxBh8OL4vdeleA0RHbB2jQKruagSfScIQ9J2zBvPxjqRqcU1882JePKTOxg8xpmUvt5z+cwxc0P6/mvkuxXi6AAqPTf//qLvIPHctsY77QZhOmor/M4kd14Xw5qJGnlrNV/P/8vQ3FXRsDOoguThNCY+03HAcp8dKTcF4M4yI3X 32lRvMUw 6AfCTqlVJDDIyYqfHFjHXnKI7zwGogervSulkkPTsKO9H07KAVwRNIuvEJNU0JzsKGr8mWX7/G9oE2ZsL8jrrPU16784r7mPA5PK9B/ELvwcwfRjrwd2JhntqfaXfzGvEMKgR8TL2qlKZ/UCUKjvOoR27WWKSfdci2AcmTu2lcAIc/gyKsiHSzvkHI+Dt/b5WAB4NMVYfYYxYFgU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It will be used to check whether the folio is mapped to specific VMA and whether the mapping address of folio is in the range. Signed-off-by: Yin Fengwei --- mm/internal.h | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/mm/internal.h b/mm/internal.h index f1276d90484ad..66117523d7d71 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -585,6 +585,32 @@ extern long faultin_vma_page_range(struct vm_area_struct *vma, bool write, int *locked); extern bool mlock_future_ok(struct mm_struct *mm, unsigned long flags, unsigned long bytes); + +static inline bool +folio_in_range(struct folio *folio, struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + pgoff_t pgoff, addr; + unsigned long vma_pglen = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT; + + VM_WARN_ON_FOLIO(folio_test_ksm(folio), folio); + if (start < vma->vm_start) + start = vma->vm_start; + + if (end > vma->vm_end) + end = vma->vm_end; + + pgoff = folio_pgoff(folio); + + /* if folio start address is not in vma range */ + if (pgoff < vma->vm_pgoff || pgoff > vma->vm_pgoff + vma_pglen) + return false; + + addr = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT); + + return ((addr >= start) && (addr + folio_size(folio) <= end)); +} + /* * mlock_vma_folio() and munlock_vma_folio(): * should be called with vma's mmap_lock held for read or write, From patchwork Fri Jul 7 16:52:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yin, Fengwei" X-Patchwork-Id: 13305124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC357EB64D9 for ; Fri, 7 Jul 2023 16:52:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6FBA56B0072; Fri, 7 Jul 2023 12:52:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6AC928D0005; Fri, 7 Jul 2023 12:52:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 573FE8D0003; Fri, 7 Jul 2023 12:52:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 496C66B0072 for ; Fri, 7 Jul 2023 12:52:53 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 127E5A0137 for ; Fri, 7 Jul 2023 16:52:53 +0000 (UTC) X-FDA: 80985410226.20.3162CE5 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf14.hostedemail.com (Postfix) with ESMTP id F331A100015 for ; Fri, 7 Jul 2023 16:52:50 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=T1Oaskz8; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf14.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688748771; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n1MYy2yizNW+bwRlKUX2YbX2UWhaWjP2YNbEtM8Q7D8=; b=0pd4dmZ54GJPiQCa2HLLk5JuErK6VuMwrUYX3NRHAWKfIGI8kX+LApviX3KEIR+8gFi8em KZVx2dB2VuKJM+UiPxy8zU868gfjzP+QN0G7CY5fsCYS+7q38mCUkdcTXkE67peCo/qqpW gh96zyhoq834oEFmNq40f1py0MNuO0c= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=T1Oaskz8; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf14.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688748771; a=rsa-sha256; cv=none; b=uBdNDrXEqttS1c3cyIVqpx9fDourbe7fhePzG+frUFdSol1fzrBA7GjHeBwNsTHomq8PPB B5miTvyT7lQ5SvdqunlLbZklCT13YsXYru3Bl9ZBy+TPeIGzVnue/ZNtJhMOjfW+uhiD0n RKEpAuagD5DRT5b0+W0uUV83pjvu6iM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688748771; x=1720284771; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hMAoCmMSZm9wJK0rzWzlbRvlWfE6c9KZJN6MJC4MnjE=; b=T1Oaskz87NjS61iNGk2uxDnML1Uw/89Jb+qVNByfZBVIPbRZAYdWuRcf jBNBBKxU8yTm/Uc8eb8xcWaxF2g61tzAWND9G/Olitp1C1diR/O/Z/usb Xxh83HLDcNN3oYGZy6d56jKdRLcQg2CvMuah/KqgpZ0Jnyn1knGlJpwEv AVac9tNo1D5HclpwUQYYkdTiMPOy8kZeP15siDb9/9iOrQ3ljKp3gVG7N xYVFhGlhpil10tUqOmkSodEitXTa/U6G7USWeLUj6xnVXI3TuZyJb0bg2 NcH6Zpx4AfTZgblSPapenEYA9I4VR7wKV3+mhpMGHNzX+RW8bsE+7p0En Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="367422316" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="367422316" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2023 09:52:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="790040971" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="790040971" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga004.fm.intel.com with ESMTP; 07 Jul 2023 09:52:46 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, yuzhao@google.com, ryan.roberts@arm.com, shy828301@gmail.com, akpm@linux-foundation.org, willy@infradead.org, david@redhat.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH 2/3] mm: handle large folio when large folio in VM_LOCKED VMA range Date: Sat, 8 Jul 2023 00:52:20 +0800 Message-Id: <20230707165221.4076590-3-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230707165221.4076590-1-fengwei.yin@intel.com> References: <20230707165221.4076590-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: F331A100015 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: hteu7xkb4jiryzrfnxzc1ynn7wgxked4 X-HE-Tag: 1688748770-112207 X-HE-Meta: U2FsdGVkX19i/69aGOo1oQ4fEoPy6H1aFcemmNtEklrwGa1pwohKn7CBIAZBMwo9/4owiIpdhXs9U0YK6NFZgHnLpXiwfvmnTzqwnDVDIX+9lLe4WFC1r7ur4CF3UoOsVwdBeRNopOunUzfGsbrRgDBrPgenAY488vmz8yMzBwI85EC3dthYKybHHMuflZAUMqTlGxECMEz8JfEfpyedcLNVYDgJ4+LJ/FVNrRHK4GJmFNMJSD9yeZ0Bw1Hn81fC4UjH2LsMtgLb72BSbccBcTzQvAwf4RZ9OpBVr2Y15TE9hDiCqRSnnPB7q/cMb8xlh7fNZxh70CAuAjSI+fwG2lOXwulGhTok7OrpbYhZzrTcdSiBeUtlttj0tN8jQbijXyeX5eP2IiJBCz2IAmjdOA3cYDuATEtslxjJCRdBGeM4ZcSpozjgGlMW+LOnKlpv6z6YF2MrXFp63K8d9XRDd4W6StRA3YkidhT0mmq6g2qQ+EIjGdGiRVOYlAQwug8IqQw1mZZno46cVmGM4BpNsTK01h/R+WgS0aEhT0cJETE8bGVHM1rffzS3fORtP6iz+haondGU+dVinDTWCELsDc55A18rEFivfsvuUGMBZB6kjl/4fIZL4WtJWrqan0LjrJH7Fo1hpgBWnuKztn1ZqKh/dMGaGvaN1Q5BK1rI7gesg1Qq3C2pEKGAoaUttrGpk3dyQ9YtRbRrWNxokvhG7rOLppv0JmWkYcW9cFEbr/VnHLRQqR8SkQ5hHsXd+A52szluXbtAcYdXmsJwxIvcA4hazsj2tbCzQCDbxfDIVX+KckFBmNihiLMpSyUc1u7KOYjVqPz1Gw+R2gFjNA8laQG0tNQG3Bef3aPWn/grsrbCpAGmxBnJpp/6g8CqT7UxRbFvlt//ehUGa3c8S8mU3N5OhcZh5M84SlJSqDds1hrihSd0MDn8ZNFj9chwp78bVylVH+R0R+JtdUJi3By 9VylwZfl 0JvWB2cvhBCcBbyibdRCY1KSXW4KjvMngQ4WE6DrLiVlbvstmFJW5l2mPDnauwbCCUc7OjmTTO+BxpPwlqsYHoVGuiMbenhAClUJFuWSveN/uTREuHc0Z4mKtUZy3hS30wNxk/IxE00jzsbEBh/o9zTpobvyoINbdOCW2/Ewm7YHTx3dpnDVg4oZErvRLZd/5OW7I3De13WvShis= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If large folio is in the range of VM_LOCKED VMA, it should be mlocked to avoid being picked by page reclaim. Which may split the large folio and then mlock each pages again. Mlock this kind of large folio to prevent them being picked by page reclaim. For the large folio which cross the boundary of VM_LOCKED VMA, we'd better not to mlock it. So if the system is under memory pressure, this kind of large folio will be split and the pages ouf of VM_LOCKED VMA can be reclaimed. Signed-off-by: Yin Fengwei --- mm/internal.h | 11 ++++++++--- mm/rmap.c | 3 ++- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 66117523d7d71..c7b8f0b008d81 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -637,7 +637,8 @@ static inline void mlock_vma_folio(struct folio *folio, * still be set while VM_SPECIAL bits are added: so ignore it then. */ if (unlikely((vma->vm_flags & (VM_LOCKED|VM_SPECIAL)) == VM_LOCKED) && - (compound || !folio_test_large(folio))) + (compound || !folio_test_large(folio) || + folio_in_range(folio, vma, vma->vm_start, vma->vm_end))) mlock_folio(folio); } @@ -645,8 +646,12 @@ void munlock_folio(struct folio *folio); static inline void munlock_vma_folio(struct folio *folio, struct vm_area_struct *vma, bool compound) { - if (unlikely(vma->vm_flags & VM_LOCKED) && - (compound || !folio_test_large(folio))) + /* + * To handle the case that a mlocked large folio is unmapped from VMA + * piece by piece, allow munlock the large folio which is partially + * mapped to VMA. + */ + if (unlikely(vma->vm_flags & VM_LOCKED)) munlock_folio(folio); } diff --git a/mm/rmap.c b/mm/rmap.c index 2668f5ea35342..7d6547d1bd096 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -817,7 +817,8 @@ static bool folio_referenced_one(struct folio *folio, address = pvmw.address; if ((vma->vm_flags & VM_LOCKED) && - (!folio_test_large(folio) || !pvmw.pte)) { + (!folio_test_large(folio) || !pvmw.pte || + folio_in_range(folio, vma, vma->vm_start, vma->vm_end))) { /* Restore the mlock which got missed */ mlock_vma_folio(folio, vma, !pvmw.pte); page_vma_mapped_walk_done(&pvmw); From patchwork Fri Jul 7 16:52:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yin, Fengwei" X-Patchwork-Id: 13305125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9109CEB64DA for ; Fri, 7 Jul 2023 16:53:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DDF36B0075; Fri, 7 Jul 2023 12:53:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B4FF8D0005; Fri, 7 Jul 2023 12:53:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17CB88D0003; Fri, 7 Jul 2023 12:53:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0A4426B0075 for ; Fri, 7 Jul 2023 12:53:09 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CF2EDB0945 for ; Fri, 7 Jul 2023 16:53:08 +0000 (UTC) X-FDA: 80985410856.24.3C37971 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf05.hostedemail.com (Postfix) with ESMTP id 5FC25100013 for ; Fri, 7 Jul 2023 16:53:05 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=SSE2sqUV; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf05.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688748786; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v6An4k3CwUFyb06/w5IRmp7Q0MPRAi6kDXl9CuGp118=; b=3Kk7bJ3pBDqBHT/3cnkhPHP/JdmtMP3kSDrkeLVyq2xEljn2bdLo3MTkMQmuED6Z+E42fD NmpDvSBgkfCMks0kW8rgcLxXsmBnQ4AULG8lPLjvEJMiTIPFqGikRQtFk+GYxlE//4I775 Vb9D8zNFtMeOtAU+9W2fnxJIt2BdApI= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=SSE2sqUV; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf05.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688748786; a=rsa-sha256; cv=none; b=lTbI6VMj5M42lEVSnReydp4NkHGMtcIE+HBotYKzygfbDIQDPsA60ORjJNzfNiBiWagssD Qh2X38MLBli1mMx0F+YyE5B973ZlRJH71xxNHZKUonoel5RxOmQctyq4j9gsGyNT99uOAF dvrhP6ZIJ3X2LzWSSkTY8V47Dg0PSn8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688748786; x=1720284786; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LCsBu7dWqfVmLSb5jMD4R7baYuKbw8fnATRe6XbxLRw=; b=SSE2sqUVwFqfJRcphl9Jncucou+1ULFh0wlZx8HWDGu3r9xxK7zFKSYD mtshjCSkSapSVxY10OZt0E3BIZgAbof1bGdfjgadPxA1sVzwwCdMhDv/A J8MTztYLls6YeojnlBNzMB5775HB2gwN2szlVyhINOxK6mc0iy+VhIT/i YLGShkJrBX8Juxlsn1RMaW1FaggU2hBVJ4t83Vv5F7Mh+30sQqBT8qLJB EeubGg3pk2lli+c9c2VMkgJ9jKxjleDfwAUeIvF2nY26V+z2yjHilD9Rd M3/RDnzXPPbALttD+2UAiB7E1v1NGXjJli3PEu7sOHUSkjDxVYko2nyPt g==; X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="427619010" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="427619010" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2023 09:53:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="966700777" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="966700777" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga006.fm.intel.com with ESMTP; 07 Jul 2023 09:52:59 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, yuzhao@google.com, ryan.roberts@arm.com, shy828301@gmail.com, akpm@linux-foundation.org, willy@infradead.org, david@redhat.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH 3/3] mm: mlock: update mlock_pte_range to handle large folio Date: Sat, 8 Jul 2023 00:52:21 +0800 Message-Id: <20230707165221.4076590-4-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230707165221.4076590-1-fengwei.yin@intel.com> References: <20230707165221.4076590-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5FC25100013 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 8dop6dkq7z1yct3xqeecg9j3eutwmghn X-HE-Tag: 1688748785-620166 X-HE-Meta: U2FsdGVkX19UYyFzjtlIoQ4P50HwkOgF0zWcfiZH+lBvi3Nu4zaFKHLoPE/4IMhXjmOxZHg+aAe60ui3EswyH6J60j7m7fcZJMwn+9izpHketRw9q2nJfsv3y9IdhQlwbIFPubX2JT/pwtd95LQm0Inp0e5zvtFg6FyresoRC47/RQlYh9mMaNuVQlHQcQ29Ir1zgrVKq9fdB26u+Y+4jBy9WX1m3lYZ9PFXtI7VrXy0a2eCBWtZ0yzV4W2D16Gzr5mAewc+bmQhQOZvH9DBJQxfbuTOOaVKWau7JqM56aXw+42OAkMjF747us6qZ4XasfmiAtLdmH0Pq5MEzqb1KpB4yzPRLNFtyCTkUAJVF5fSttfPRR8sttqcFdGZWKIhT/oCX9LFeZaPrdwk9bv4wpA9ChRtFDNloVY2MvBwL17EUowPgASH0Sg0tP3PldJYQLFs4avx1u/DxJAWeBrI2kq4TMHDGln41rSdSC6dSSEhPvcIIMTwWlq5K7FAkGpLXcGhJTQpniUxZFGBwWjmW18YTOqM++FKKfcsmWdmfFajwhNPb2p7iwoi4yppMIMK/FX/QQ/cDUKCJ0lE0O4pKMnyor14i70AHSrBqdgnFFC+DYBkavm2dRCAQevquEUXkaM/H1Rtf9viq6kS11SJmCGFoeYF0aQvSppIzDYQWOnMP5T8+bMVkV1F9Rf+lapXttIbOQDN22+hKNbU0VatWQns7zmcalrRdZosiiaL5NjxDuc/5cMxSVJMhRT6gC7uneUA7WZCjzfuc8QhQ7s4vIhd1ULkvp3EoHKTZcqES08HlGcbq/NrIvN3eH6+86zhglTD5BkVEYHiE8nBTTmMzxKu3+Ym7/HU5Q7t5ikykazzdXMzPZzizVF4fFIH9CPQ1ZIAUnH2N3qi3pj3ps+LU9GKzTHUIz8+rsESImbKnNobkYD+JBU+fXUfhToF3Hc06IlVxI3/wK3y/5TeXmI IWVrR4xv dgH4npAtpXL7gmuJNJbvhlFvr2RxuEsjaI7ebI+TguO9ttn8uOIdYs+KeRZaspxCt65cJx2qtyEzq9sYxYp2X+FJeaBowU/uj05DvuEtYUTElI6Pw+Q+KYCQE98tq3UUuxG35NejkbuamPNmJvQFQbbSP3VnJ5ifeiYLdFuEzDAM0fzYqKj2kQrLoQDlh+nXeTvOE7GDOZpxU5FU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Current kernel only lock base size folio during mlock syscall. Add large folio support with following rules: - Only mlock large folio when it's in VM_LOCKED VMA range - If there is cow folio, mlock the cow folio as cow folio is also in VM_LOCKED VMA range. - munlock will apply to the large folio which is in VMA range or cross the VMA boundary. The last rule is used to handle the case that the large folio is mlocked, later the VMA is split in the middle of large folio and this large folio become cross VMA boundary. Signed-off-by: Yin Fengwei --- mm/mlock.c | 103 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 95 insertions(+), 8 deletions(-) diff --git a/mm/mlock.c b/mm/mlock.c index d7db94519884d..e09a481062ef5 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -305,6 +305,64 @@ void munlock_folio(struct folio *folio) local_unlock(&mlock_fbatch.lock); } +void mlock_folio_range(struct folio *folio, struct vm_area_struct *vma, + pte_t *pte, unsigned long addr, unsigned int nr) +{ + struct folio *cow_folio; + unsigned int step = 1; + + mlock_folio(folio); + if (nr == 1) + return; + + for (; nr > 0; pte += step, addr += (step << PAGE_SHIFT), nr -= step) { + pte_t ptent; + + step = 1; + ptent = ptep_get(pte); + + if (!pte_present(ptent)) + continue; + + cow_folio = vm_normal_folio(vma, addr, ptent); + if (!cow_folio || cow_folio == folio) { + continue; + } + + mlock_folio(cow_folio); + step = min_t(unsigned int, nr, folio_nr_pages(cow_folio)); + } +} + +void munlock_folio_range(struct folio *folio, struct vm_area_struct *vma, + pte_t *pte, unsigned long addr, unsigned int nr) +{ + struct folio *cow_folio; + unsigned int step = 1; + + munlock_folio(folio); + if (nr == 1) + return; + + for (; nr > 0; pte += step, addr += (step << PAGE_SHIFT), nr -= step) { + pte_t ptent; + + step = 1; + ptent = ptep_get(pte); + + if (!pte_present(ptent)) + continue; + + cow_folio = vm_normal_folio(vma, addr, ptent); + if (!cow_folio || cow_folio == folio) { + continue; + } + + munlock_folio(cow_folio); + step = min_t(unsigned int, nr, folio_nr_pages(cow_folio)); + } +} + static int mlock_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -314,6 +372,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, pte_t *start_pte, *pte; pte_t ptent; struct folio *folio; + unsigned int step = 1, nr; ptl = pmd_trans_huge_lock(pmd, vma); if (ptl) { @@ -329,24 +388,52 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, goto out; } - start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + pte = start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); if (!start_pte) { walk->action = ACTION_AGAIN; return 0; } - for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { + + for (; addr != end; pte += step, addr += (step << PAGE_SHIFT)) { + step = 1; ptent = ptep_get(pte); if (!pte_present(ptent)) continue; folio = vm_normal_folio(vma, addr, ptent); if (!folio || folio_is_zone_device(folio)) continue; - if (folio_test_large(folio)) - continue; - if (vma->vm_flags & VM_LOCKED) - mlock_folio(folio); - else - munlock_folio(folio); + + folio_get(folio); + nr = folio_nr_pages(folio) + folio_pfn(folio) - pte_pfn(ptent); + nr = min_t(unsigned int, nr, (end - addr) >> PAGE_SHIFT); + + if (vma->vm_flags & VM_LOCKED) { + /* + * Only mlock the 4K folio or large folio + * in VMA range + */ + if (folio_test_large(folio) && + !folio_in_range(folio, vma, + vma->vm_start, vma->vm_end)) { + folio_put(folio); + continue; + } + mlock_folio_range(folio, vma, pte, addr, nr); + } else { + /* + * Allow munlock large folio which is partially mapped + * to VMA. As it's possible that large folio is mlocked + * and VMA is split later. + * + * During memory pressure, such kind of large folio can + * be split. And the pages are not in VM_LOCKed VMA + * can be reclaimed. + */ + munlock_folio_range(folio, vma, pte, addr, nr); + } + + step = nr; + folio_put(folio); } pte_unmap(start_pte); out: