From patchwork Tue Mar 4 15:04:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 14000869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0DC0C021B8 for ; Tue, 4 Mar 2025 15:05:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B82176B0093; Tue, 4 Mar 2025 10:05:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B2F056B0095; Tue, 4 Mar 2025 10:05:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9AAF26B0096; Tue, 4 Mar 2025 10:05:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 72EC06B0093 for ; Tue, 4 Mar 2025 10:05:08 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0FF774B569 for ; Tue, 4 Mar 2025 15:05:08 +0000 (UTC) X-FDA: 83184191496.19.25536F9 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf08.hostedemail.com (Postfix) with ESMTP id 6F3EC16000B for ; Tue, 4 Mar 2025 15:05:06 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf08.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741100706; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TF/6uGlm1+JnFATcSTMQMNzWrve0WLCN606nc3w+F2M=; b=2usR/mfMznz7bhOZ+8LI31FB4Q14EXgKqG/dU1cDS0Z0ovZ7SkV90y4kaZFG0bTae2dSOJ mhn75kUXqPcmG9mGidvcIv0Y9QBoqs61eAGzrUgD7RMjITeJJBL0Lo0Vx+HVG/aMrsQjyf A4tLA9aubjxeZvJKLMPZWbBY+0X5L+Q= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf08.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741100706; a=rsa-sha256; cv=none; b=QvjOxq3vbrryBLx8eJwqCGlo8YHlBKF7tmYFKqXJhHuCxKKQOVwcFWEaz980uarVOPZ/wK AskmTp8shNXN5/3LOz7jLfbtP2oR4/SpCui5bB3SQs/py0bKeclGw/u8cyQazZG/+Gm0KO 0Fc7fogDUxWncN5t4vpVNYDZg61c/r8= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 75EFD1C2B; Tue, 4 Mar 2025 07:05:19 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E43683F66E; Tue, 4 Mar 2025 07:05:03 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , David Hildenbrand , "Matthew Wilcox (Oracle)" , Mark Rutland , Anshuman Khandual , Alexandre Ghiti , Kevin Brodsky Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 05/11] arm64: hugetlb: Use set_ptes_anysz() and ptep_get_and_clear_anysz() Date: Tue, 4 Mar 2025 15:04:35 +0000 Message-ID: <20250304150444.3788920-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250304150444.3788920-1-ryan.roberts@arm.com> References: <20250304150444.3788920-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6F3EC16000B X-Stat-Signature: wjzs9gbuhkbq4115x18cuioigejgw6a6 X-HE-Tag: 1741100706-157516 X-HE-Meta: U2FsdGVkX1/eceKzm8exW7hcLAsr+ZDYhQumUSzqDM0gM3jpRMSHy16P979DnAdFkyt6r4RIKtMoLRIdq3NkXCMbREQF863AsMCcuQSwATTkws63UKupsahehqkrxSAvNdO5U1KCyhEs6aU0X+FZP6xPMfU+JZnypViOokobPXzgzngE/dbIKQsTlkyhuDaeUG4sqahxkPCUMj2VaLIxVEDGkp32Gp7f8iyKJY5TcwfcPWtKhSHm+xNKWfEM3mcgIKPrj+TYrmLxCKwRgb4fGeTvO9Rynh14k5qcQUOIleGIx76ZT9rGl1YFyC5YEOY0aczMFUJ427dC4PIu9Jl6OBfMU4DVzPxSdYyfxkHSPfpFf4ShNmlqTLzD0JnFWJfFAUnoJkfq3SFXnHILC/d26G5uz5rwOAjJcWUZFZn0X/eKcBFdN4uDdvHV/ZCHXQ4TZGNQVNLuQlUQNxiSFeoP3EKq1zKg69fGzFNFWoCRcd2r6o27CQ+WaNLAK/PzCiv4lcOqvBhK1llkm9v/AUfTmntYcGL8FIbAKqRE0S4cTyjnmErmKDMzRgDn6PMzPL0ULuIFFw/ym6VUjiUcFKOKOZOaA7Jdzgs4i5LFFNL7LSVSXRpMHW/nu4aRQ+QoXjyIMqI02A1CKhC7iZ51gk4iurYEkpzS74F4YWAgEdX5D9ETfHThL3iyyNfjzHKQ/U+5oAQyH0Q27u4T76e3MIm3/eXmHO0lcjm//gYSfu5q2FVMIc/hVX6a/T8KNwiXsZVbBGUCJ6IwCT1phEFlmBsUgvnSWKUKjqxBJiCFTpVfBTHJABr9NfWwyeaIG9avksCAqneGEGmGBe2OhgBf7HV32bsr1YVp5LQabf22u0s4qx7cgUgFafM/NsfTucwjPE9gEWXmG1hv7a/Qf8ItTkBNCqRx3YQEB1iFd7d+L5iCSOPwrgKa+rAZJCte+KvQAu6A7WOMfUSFFfjAJciiDdh bdMnoBLm 6iiD4zhebNdwgcyFli/h8JjE/mgzeLbqzfW+vlGiHi0uGCu48BStSeztbmVSmYE7soJaXc7COnG8FQ9YQHJY4x8HcaWdEnpdo5KSXkcWauIM+YL9kvjn3ni9jSCviXebMl6SH2R7Svg+/8JD8xXqsgL5A4ekPfrjGBps+mnEbOfGFM4KUP62zwS4i07w2mUlOEGO0EMsCLkncA64= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refactor the huge_pte helpers to use the new common set_ptes_anysz() and ptep_get_and_clear_anysz() APIs. This provides 2 benefits; First, when page_table_check=on, hugetlb is now properly/fully checked. Previously only the first page of a hugetlb folio was checked. Second, instead of having to call __set_ptes(nr=1) for each pte in a loop, the whole contiguous batch can now be set in one go, which enables some efficiencies and cleans up the code. One detail to note is that huge_ptep_clear_flush() was previously calling ptep_clear_flush() for a non-contiguous pte (i.e. a pud or pmd block mapping). This has a couple of disadvantages; first ptep_clear_flush() calls ptep_get_and_clear() which transparently handles contpte. Given we only call for non-contiguous ptes, it would be safe, but a waste of effort. It's preferable to go straight to the layer below. However, more problematic is that ptep_get_and_clear() is for PAGE_SIZE entries so it calls page_table_check_pte_clear() and would not clear the whole hugetlb folio. So let's stop special-casing the non-cont case and just rely on get_clear_contig_flush() to do the right thing for non-cont entries. Signed-off-by: Ryan Roberts --- arch/arm64/mm/hugetlbpage.c | 52 +++++++------------------------------ 1 file changed, 10 insertions(+), 42 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 065be8650aa5..efd18bd1eae3 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -159,12 +159,12 @@ static pte_t get_clear_contig(struct mm_struct *mm, pte_t pte, tmp_pte; bool present; - pte = __ptep_get_and_clear(mm, addr, ptep); + pte = ptep_get_and_clear_anysz(mm, ptep, pgsize); present = pte_present(pte); while (--ncontig) { ptep++; addr += pgsize; - tmp_pte = __ptep_get_and_clear(mm, addr, ptep); + tmp_pte = ptep_get_and_clear_anysz(mm, ptep, pgsize); if (present) { if (pte_dirty(tmp_pte)) pte = pte_mkdirty(pte); @@ -208,7 +208,7 @@ static void clear_flush(struct mm_struct *mm, unsigned long i, saddr = addr; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - __ptep_get_and_clear(mm, addr, ptep); + ptep_get_and_clear_anysz(mm, ptep, pgsize); __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, true); } @@ -219,32 +219,20 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, size_t pgsize; int i; int ncontig; - unsigned long pfn, dpfn; - pgprot_t hugeprot; ncontig = num_contig_ptes(sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - __set_ptes(mm, addr, ptep, pte, 1); + set_ptes_anysz(mm, ptep, pte, 1, pgsize); return; } - if (!pte_cont(pte)) { - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - pfn = pte_pfn(pte); - dpfn = pgsize >> PAGE_SHIFT; - hugeprot = pte_pgprot(pte); - /* Only need to "break" if transitioning valid -> valid. */ - if (pte_valid(__ptep_get(ptep))) + if (pte_cont(pte) && pte_valid(__ptep_get(ptep))) clear_flush(mm, addr, ptep, pgsize, ncontig); - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); + set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); } pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, @@ -434,11 +422,9 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty) { - int ncontig, i; + int ncontig; size_t pgsize = 0; - unsigned long pfn = pte_pfn(pte), dpfn; struct mm_struct *mm = vma->vm_mm; - pgprot_t hugeprot; pte_t orig_pte; VM_WARN_ON(!pte_present(pte)); @@ -447,7 +433,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); ncontig = num_contig_ptes(huge_page_size(hstate_vma(vma)), &pgsize); - dpfn = pgsize >> PAGE_SHIFT; if (!__cont_access_flags_changed(ptep, pte, ncontig)) return 0; @@ -462,19 +447,14 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (pte_young(orig_pte)) pte = pte_mkyoung(pte); - hugeprot = pte_pgprot(pte); - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); - + set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); return 1; } void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - unsigned long pfn, dpfn; - pgprot_t hugeprot; - int ncontig, i; + int ncontig; size_t pgsize; pte_t pte; @@ -487,16 +467,11 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, } ncontig = find_num_contig(mm, addr, ptep, &pgsize); - dpfn = pgsize >> PAGE_SHIFT; pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); pte = pte_wrprotect(pte); - hugeprot = pte_pgprot(pte); - pfn = pte_pfn(pte); - - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); + set_ptes_anysz(mm, ptep, pte, ncontig, pgsize); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, @@ -505,13 +480,6 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; size_t pgsize; int ncontig; - pte_t pte; - - pte = __ptep_get(ptep); - VM_WARN_ON(!pte_present(pte)); - - if (!pte_cont(pte)) - return ptep_clear_flush(vma, addr, ptep); ncontig = num_contig_ptes(huge_page_size(hstate_vma(vma)), &pgsize); return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig);