From patchwork Fri Jul 7 16:52:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yin, Fengwei" X-Patchwork-Id: 13305125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9109CEB64DA for ; Fri, 7 Jul 2023 16:53:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DDF36B0075; Fri, 7 Jul 2023 12:53:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B4FF8D0005; Fri, 7 Jul 2023 12:53:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 17CB88D0003; Fri, 7 Jul 2023 12:53:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0A4426B0075 for ; Fri, 7 Jul 2023 12:53:09 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CF2EDB0945 for ; Fri, 7 Jul 2023 16:53:08 +0000 (UTC) X-FDA: 80985410856.24.3C37971 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by imf05.hostedemail.com (Postfix) with ESMTP id 5FC25100013 for ; Fri, 7 Jul 2023 16:53:05 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=SSE2sqUV; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf05.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688748786; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v6An4k3CwUFyb06/w5IRmp7Q0MPRAi6kDXl9CuGp118=; b=3Kk7bJ3pBDqBHT/3cnkhPHP/JdmtMP3kSDrkeLVyq2xEljn2bdLo3MTkMQmuED6Z+E42fD NmpDvSBgkfCMks0kW8rgcLxXsmBnQ4AULG8lPLjvEJMiTIPFqGikRQtFk+GYxlE//4I775 Vb9D8zNFtMeOtAU+9W2fnxJIt2BdApI= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=SSE2sqUV; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf05.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.31 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688748786; a=rsa-sha256; cv=none; b=lTbI6VMj5M42lEVSnReydp4NkHGMtcIE+HBotYKzygfbDIQDPsA60ORjJNzfNiBiWagssD Qh2X38MLBli1mMx0F+YyE5B973ZlRJH71xxNHZKUonoel5RxOmQctyq4j9gsGyNT99uOAF dvrhP6ZIJ3X2LzWSSkTY8V47Dg0PSn8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688748786; x=1720284786; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LCsBu7dWqfVmLSb5jMD4R7baYuKbw8fnATRe6XbxLRw=; b=SSE2sqUVwFqfJRcphl9Jncucou+1ULFh0wlZx8HWDGu3r9xxK7zFKSYD mtshjCSkSapSVxY10OZt0E3BIZgAbof1bGdfjgadPxA1sVzwwCdMhDv/A J8MTztYLls6YeojnlBNzMB5775HB2gwN2szlVyhINOxK6mc0iy+VhIT/i YLGShkJrBX8Juxlsn1RMaW1FaggU2hBVJ4t83Vv5F7Mh+30sQqBT8qLJB EeubGg3pk2lli+c9c2VMkgJ9jKxjleDfwAUeIvF2nY26V+z2yjHilD9Rd M3/RDnzXPPbALttD+2UAiB7E1v1NGXjJli3PEu7sOHUSkjDxVYko2nyPt g==; X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="427619010" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="427619010" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2023 09:53:03 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10764"; a="966700777" X-IronPort-AV: E=Sophos;i="6.01,189,1684825200"; d="scan'208";a="966700777" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by fmsmga006.fm.intel.com with ESMTP; 07 Jul 2023 09:52:59 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, yuzhao@google.com, ryan.roberts@arm.com, shy828301@gmail.com, akpm@linux-foundation.org, willy@infradead.org, david@redhat.com Cc: fengwei.yin@intel.com Subject: [RFC PATCH 3/3] mm: mlock: update mlock_pte_range to handle large folio Date: Sat, 8 Jul 2023 00:52:21 +0800 Message-Id: <20230707165221.4076590-4-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230707165221.4076590-1-fengwei.yin@intel.com> References: <20230707165221.4076590-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5FC25100013 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 8dop6dkq7z1yct3xqeecg9j3eutwmghn X-HE-Tag: 1688748785-620166 X-HE-Meta: U2FsdGVkX19UYyFzjtlIoQ4P50HwkOgF0zWcfiZH+lBvi3Nu4zaFKHLoPE/4IMhXjmOxZHg+aAe60ui3EswyH6J60j7m7fcZJMwn+9izpHketRw9q2nJfsv3y9IdhQlwbIFPubX2JT/pwtd95LQm0Inp0e5zvtFg6FyresoRC47/RQlYh9mMaNuVQlHQcQ29Ir1zgrVKq9fdB26u+Y+4jBy9WX1m3lYZ9PFXtI7VrXy0a2eCBWtZ0yzV4W2D16Gzr5mAewc+bmQhQOZvH9DBJQxfbuTOOaVKWau7JqM56aXw+42OAkMjF747us6qZ4XasfmiAtLdmH0Pq5MEzqb1KpB4yzPRLNFtyCTkUAJVF5fSttfPRR8sttqcFdGZWKIhT/oCX9LFeZaPrdwk9bv4wpA9ChRtFDNloVY2MvBwL17EUowPgASH0Sg0tP3PldJYQLFs4avx1u/DxJAWeBrI2kq4TMHDGln41rSdSC6dSSEhPvcIIMTwWlq5K7FAkGpLXcGhJTQpniUxZFGBwWjmW18YTOqM++FKKfcsmWdmfFajwhNPb2p7iwoi4yppMIMK/FX/QQ/cDUKCJ0lE0O4pKMnyor14i70AHSrBqdgnFFC+DYBkavm2dRCAQevquEUXkaM/H1Rtf9viq6kS11SJmCGFoeYF0aQvSppIzDYQWOnMP5T8+bMVkV1F9Rf+lapXttIbOQDN22+hKNbU0VatWQns7zmcalrRdZosiiaL5NjxDuc/5cMxSVJMhRT6gC7uneUA7WZCjzfuc8QhQ7s4vIhd1ULkvp3EoHKTZcqES08HlGcbq/NrIvN3eH6+86zhglTD5BkVEYHiE8nBTTmMzxKu3+Ym7/HU5Q7t5ikykazzdXMzPZzizVF4fFIH9CPQ1ZIAUnH2N3qi3pj3ps+LU9GKzTHUIz8+rsESImbKnNobkYD+JBU+fXUfhToF3Hc06IlVxI3/wK3y/5TeXmI IWVrR4xv dgH4npAtpXL7gmuJNJbvhlFvr2RxuEsjaI7ebI+TguO9ttn8uOIdYs+KeRZaspxCt65cJx2qtyEzq9sYxYp2X+FJeaBowU/uj05DvuEtYUTElI6Pw+Q+KYCQE98tq3UUuxG35NejkbuamPNmJvQFQbbSP3VnJ5ifeiYLdFuEzDAM0fzYqKj2kQrLoQDlh+nXeTvOE7GDOZpxU5FU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Current kernel only lock base size folio during mlock syscall. Add large folio support with following rules: - Only mlock large folio when it's in VM_LOCKED VMA range - If there is cow folio, mlock the cow folio as cow folio is also in VM_LOCKED VMA range. - munlock will apply to the large folio which is in VMA range or cross the VMA boundary. The last rule is used to handle the case that the large folio is mlocked, later the VMA is split in the middle of large folio and this large folio become cross VMA boundary. Signed-off-by: Yin Fengwei --- mm/mlock.c | 103 ++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 95 insertions(+), 8 deletions(-) diff --git a/mm/mlock.c b/mm/mlock.c index d7db94519884d..e09a481062ef5 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -305,6 +305,64 @@ void munlock_folio(struct folio *folio) local_unlock(&mlock_fbatch.lock); } +void mlock_folio_range(struct folio *folio, struct vm_area_struct *vma, + pte_t *pte, unsigned long addr, unsigned int nr) +{ + struct folio *cow_folio; + unsigned int step = 1; + + mlock_folio(folio); + if (nr == 1) + return; + + for (; nr > 0; pte += step, addr += (step << PAGE_SHIFT), nr -= step) { + pte_t ptent; + + step = 1; + ptent = ptep_get(pte); + + if (!pte_present(ptent)) + continue; + + cow_folio = vm_normal_folio(vma, addr, ptent); + if (!cow_folio || cow_folio == folio) { + continue; + } + + mlock_folio(cow_folio); + step = min_t(unsigned int, nr, folio_nr_pages(cow_folio)); + } +} + +void munlock_folio_range(struct folio *folio, struct vm_area_struct *vma, + pte_t *pte, unsigned long addr, unsigned int nr) +{ + struct folio *cow_folio; + unsigned int step = 1; + + munlock_folio(folio); + if (nr == 1) + return; + + for (; nr > 0; pte += step, addr += (step << PAGE_SHIFT), nr -= step) { + pte_t ptent; + + step = 1; + ptent = ptep_get(pte); + + if (!pte_present(ptent)) + continue; + + cow_folio = vm_normal_folio(vma, addr, ptent); + if (!cow_folio || cow_folio == folio) { + continue; + } + + munlock_folio(cow_folio); + step = min_t(unsigned int, nr, folio_nr_pages(cow_folio)); + } +} + static int mlock_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -314,6 +372,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, pte_t *start_pte, *pte; pte_t ptent; struct folio *folio; + unsigned int step = 1, nr; ptl = pmd_trans_huge_lock(pmd, vma); if (ptl) { @@ -329,24 +388,52 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, goto out; } - start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + pte = start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); if (!start_pte) { walk->action = ACTION_AGAIN; return 0; } - for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { + + for (; addr != end; pte += step, addr += (step << PAGE_SHIFT)) { + step = 1; ptent = ptep_get(pte); if (!pte_present(ptent)) continue; folio = vm_normal_folio(vma, addr, ptent); if (!folio || folio_is_zone_device(folio)) continue; - if (folio_test_large(folio)) - continue; - if (vma->vm_flags & VM_LOCKED) - mlock_folio(folio); - else - munlock_folio(folio); + + folio_get(folio); + nr = folio_nr_pages(folio) + folio_pfn(folio) - pte_pfn(ptent); + nr = min_t(unsigned int, nr, (end - addr) >> PAGE_SHIFT); + + if (vma->vm_flags & VM_LOCKED) { + /* + * Only mlock the 4K folio or large folio + * in VMA range + */ + if (folio_test_large(folio) && + !folio_in_range(folio, vma, + vma->vm_start, vma->vm_end)) { + folio_put(folio); + continue; + } + mlock_folio_range(folio, vma, pte, addr, nr); + } else { + /* + * Allow munlock large folio which is partially mapped + * to VMA. As it's possible that large folio is mlocked + * and VMA is split later. + * + * During memory pressure, such kind of large folio can + * be split. And the pages are not in VM_LOCKed VMA + * can be reclaimed. + */ + munlock_folio_range(folio, vma, pte, addr, nr); + } + + step = nr; + folio_put(folio); } pte_unmap(start_pte); out: