From patchwork Mon Dec 4 10:54:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92576C4167B for ; Mon, 4 Dec 2023 10:55:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F1F1A6B029A; Mon, 4 Dec 2023 05:55:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ECE546B029D; Mon, 4 Dec 2023 05:55:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D70386B029E; Mon, 4 Dec 2023 05:55:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C52886B029A for ; Mon, 4 Dec 2023 05:55:02 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 97B7D1A018D for ; Mon, 4 Dec 2023 10:55:02 +0000 (UTC) X-FDA: 81528828444.15.B467ABE Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf05.hostedemail.com (Postfix) with ESMTP id E7392100006 for ; Mon, 4 Dec 2023 10:55:00 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf05.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687301; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jMOlX8fCWcEgenegWdNgg1CacU1memV2d2wH88+tx58=; b=ICJPnkfvW564lo18saXEKPgD4Nh9mjhpS0ZYNJu0gLIoH26yJ9mLwQkLblMhWmiQ9pG24M TOAj8V4mhJ5EK2HrcvHh7FH0z2ZynORG87setfKR+ynqv3dKMSs87EYxxCBngCXVx9398T 7avKZ4SWjNAKY8sh9nBLEOs8MtF6MEo= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf05.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687301; a=rsa-sha256; cv=none; b=FUGstsVuf+HBeTjL1yEHiYJrUp/OXR7mSGqe10QimnbpFEw9LdjUVn0gsbGNYBthpWGMPe dWHcUTuDbaVHWujpuskpGjSi5/kyKkKAw9gPks3hzmNHPq3tO+EsZ0CuLpUmELg8avJC7a aUVXSqdC2jfwkCvRZc/glXjN+9BPQgg= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 76D1A1684; Mon, 4 Dec 2023 02:55:47 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A82483F6C4; Mon, 4 Dec 2023 02:54:56 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 01/15] mm: Batch-copy PTE ranges during fork() Date: Mon, 4 Dec 2023 10:54:26 +0000 Message-Id: <20231204105440.61448-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E7392100006 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: a5sd5kdkirji89w4ypp9dn77k7pzeiem X-HE-Tag: 1701687300-266439 X-HE-Meta: U2FsdGVkX18VbrA/4TdkxjYTHmkQVpIlngYpfuoMisXK5QszI2HMVnuyvS4mE8qnuAfm2b4MGDO8aNWkejZHvDlOCtVTi70iAVpXkwEkz0CHnp0i7d8CNzRq3wN5gnqUU7wPkweYFjdd2Yo3C1BdQS7/WfMfTtfYIy/7PoKcFdbxGsGgPV873tt2yKouLdBEMl/ehARFe5OyKsY/IaKe+ay5K+cKTIV7HM49uCFKfDqv5Au6dPLZIDkdoQnhAb6jY0QihEdVhHVrU+7/gRf3GYn5BM9u4I8XvMMuT9vKz4WNrCRmJZ3A25rlZ4J1EFrOypj7VRvJK8AA2WtbFGx95YTzSqU9al9oDNhRurYHcUmkgLwMRax083yB+B64thg8APYRNVx8vURd98dxmAr2K8fenXj32Uo+GLojW/jqGE1AVGW+kRl/1ZUYXx7cR3bPllMWlLVv5WFWjcp6rnDrDNaKhxiVB+XwPhkundfVZpHX9+zlCaHyFiZRSadJ6ARBOQR5FBJ9lhM/P65mWMCr8OunVJhusD3pHqCwovn2+7PoaZrvRs1d09CQi5YYUxNYLwjj6DrAiZHAfmYlrvCg5Aua65g4uEaif/G9LUk3Yt3keVJrEE5d1CiBeVxh/g2110Rp1byazxKBWQeHbQP/dpyIOTYwotFafaFd2AZiebnAtzarJbp9jPNvk6FZYY0YY3q+lddFak4Khyd6ydg6VjWeT17bYE03vUsyJUGw1vdLtijghmiN1GuHW6986txAKfitiDpeoHuH6pNdnUfboFN4lq5CIkID61LbqwabgzHkgT1QfHYXtTREexK2xn9KMO06BEUoXJERmiSuD6x8wcMYv5VLOWgEAXDwjyLl4z7p9LUoAxIpbu3U0HRg/fHVn79NNOs9ZWPlvtjG19WDai62HWv203eL7N5h4zV2T81tYGz/ZrEAU6vRQlBpxa81n36RIC+uumcuJR7RlFu BF2nAYC/ z8jDkAQxDotBPTicu+Gd/i1YCf+B8S7tJiJY0wC1zZu9WNaXn/+X/hxerf1wZfD2HXQVh+Ste3yx7ynxKpVXUSFKYqrzBA3jlZ1hDGu4j/rf0K5beHJReJxq7L97youdiYIEDuZarwF6iNe0oTnj8WNSZ0dhjtPoDSjwaQr+lAWIF8Vnns4Sp6amli+BAX8nkFWl/XOpzAtlJmv9b5YIZkdezfYdJf7QKWs1EUILVN0VCwoQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Convert copy_pte_range() to copy a set of ptes in a batch. A given batch maps a physically contiguous block of memory, all belonging to the same folio. This will likely improve performance by a tiny amount due to batching the folio reference count management and calling set_ptes() rather than making individual calls to set_pte_at(). However, the primary motivation for this change is to reduce the number of tlb maintenance operations that the arm64 backend has to perform during fork, as it is about to add transparent support for the "contiguous bit" in its ptes. By write-protecting the parent using the new ptep_set_wrprotects() (note the 's' at the end) function, the backend can avoid having to unfold contig ranges of PTEs, which is expensive, when all ptes in the range are being write-protected. Similarly, by using set_ptes() rather than set_pte_at() to set up ptes in the child, the backend does not need to fold a contiguous range once they are all populated - they can be initially populated as a contiguous range in the first place. This change addresses the core-mm refactoring only, and introduces ptep_set_wrprotects() with a default implementation that calls ptep_set_wrprotect() for each pte in the range. A separate change will implement ptep_set_wrprotects() in the arm64 backend to realize the performance improvement as part of the work to enable contpte mappings. Signed-off-by: Ryan Roberts Reviewed-by: Alistair Popple --- include/linux/pgtable.h | 13 +++ mm/memory.c | 195 ++++++++++++++++++++++++++++++---------- 2 files changed, 162 insertions(+), 46 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index af7639c3b0a3..1c50f8a0fdde 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -622,6 +622,19 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres } #endif +#ifndef ptep_set_wrprotects +struct mm_struct; +static inline void ptep_set_wrprotects(struct mm_struct *mm, + unsigned long address, pte_t *ptep, + unsigned int nr) +{ + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + ptep_set_wrprotect(mm, address, ptep); +} +#endif + /* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings diff --git a/mm/memory.c b/mm/memory.c index 1f18ed4a5497..8a87a488950c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -924,68 +924,162 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma return 0; } +static int folio_nr_pages_cont_mapped(struct folio *folio, + struct page *page, pte_t *pte, + unsigned long addr, unsigned long end, + pte_t ptent, bool enforce_uffd_wp, + int *dirty_nr, int *writable_nr) +{ + int floops; + int i; + unsigned long pfn; + bool prot_none; + bool uffd_wp; + + if (!folio_test_large(folio)) + return 1; + + /* + * Loop either to `end` or to end of folio if its contiguously mapped, + * whichever is smaller. + */ + floops = (end - addr) >> PAGE_SHIFT; + floops = min_t(int, floops, + folio_pfn(folio_next(folio)) - page_to_pfn(page)); + + pfn = page_to_pfn(page); + prot_none = pte_protnone(ptent); + uffd_wp = pte_uffd_wp(ptent); + + *dirty_nr = !!pte_dirty(ptent); + *writable_nr = !!pte_write(ptent); + + pfn++; + pte++; + + for (i = 1; i < floops; i++) { + ptent = ptep_get(pte); + + if (!pte_present(ptent) || pte_pfn(ptent) != pfn || + prot_none != pte_protnone(ptent) || + (enforce_uffd_wp && uffd_wp != pte_uffd_wp(ptent))) + break; + + if (pte_dirty(ptent)) + (*dirty_nr)++; + if (pte_write(ptent)) + (*writable_nr)++; + + pfn++; + pte++; + } + + return i; +} + /* - * Copy one pte. Returns 0 if succeeded, or -EAGAIN if one preallocated page - * is required to copy this pte. + * Copy set of contiguous ptes. Returns number of ptes copied if succeeded + * (always gte 1), or -EAGAIN if one preallocated page is required to copy the + * first pte. */ static inline int -copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, - pte_t *dst_pte, pte_t *src_pte, unsigned long addr, int *rss, - struct folio **prealloc) +copy_present_ptes(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, + pte_t *dst_pte, pte_t *src_pte, + unsigned long addr, unsigned long end, + int *rss, struct folio **prealloc) { struct mm_struct *src_mm = src_vma->vm_mm; unsigned long vm_flags = src_vma->vm_flags; pte_t pte = ptep_get(src_pte); struct page *page; struct folio *folio; + int nr = 1; + bool anon = false; + bool enforce_uffd_wp = userfaultfd_wp(dst_vma); + int nr_dirty = !!pte_dirty(pte); + int nr_writable = !!pte_write(pte); + int i, ret; page = vm_normal_page(src_vma, addr, pte); - if (page) + if (page) { folio = page_folio(page); - if (page && folio_test_anon(folio)) { - /* - * If this page may have been pinned by the parent process, - * copy the page immediately for the child so that we'll always - * guarantee the pinned page won't be randomly replaced in the - * future. - */ - folio_get(folio); - if (unlikely(page_try_dup_anon_rmap(page, false, src_vma))) { - /* Page may be pinned, we have to copy. */ - folio_put(folio); - return copy_present_page(dst_vma, src_vma, dst_pte, src_pte, - addr, rss, prealloc, page); + anon = folio_test_anon(folio); + nr = folio_nr_pages_cont_mapped(folio, page, src_pte, addr, end, + pte, enforce_uffd_wp, &nr_dirty, + &nr_writable); + folio_ref_add(folio, nr); + + for (i = 0; i < nr; i++, page++) { + if (anon) { + /* + * If this page may have been pinned by the + * parent process, copy the page immediately for + * the child so that we'll always guarantee the + * pinned page won't be randomly replaced in the + * future. + */ + if (unlikely(page_try_dup_anon_rmap( + page, false, src_vma))) { + if (i != 0) + break; + /* Page may be pinned, we have to copy. */ + folio_ref_sub(folio, nr); + ret = copy_present_page( + dst_vma, src_vma, dst_pte, + src_pte, addr, rss, prealloc, + page); + return ret == 0 ? 1 : ret; + } + rss[MM_ANONPAGES]++; + VM_BUG_ON(PageAnonExclusive(page)); + } else { + page_dup_file_rmap(page, false); + rss[mm_counter_file(page)]++; + } } - rss[MM_ANONPAGES]++; - } else if (page) { - folio_get(folio); - page_dup_file_rmap(page, false); - rss[mm_counter_file(page)]++; - } - /* - * If it's a COW mapping, write protect it both - * in the parent and the child - */ - if (is_cow_mapping(vm_flags) && pte_write(pte)) { - ptep_set_wrprotect(src_mm, addr, src_pte); - pte = pte_wrprotect(pte); + if (i < nr) { + folio_ref_sub(folio, nr - i); + nr = i; + } } - VM_BUG_ON(page && folio_test_anon(folio) && PageAnonExclusive(page)); /* - * If it's a shared mapping, mark it clean in - * the child + * If it's a shared mapping, mark it clean and write protected in the + * child, and rely on a write fault to fix up the permissions. This + * allows determining batch size without having to consider RO/RW + * permissions. As an optimization, skip wrprotect if all ptes in the + * batch have the same permissions. + * + * If its a private (CoW) mapping, mark it dirty in the child if _any_ + * of the parent mappings in the block were marked dirty. The contiguous + * block of mappings are all backed by the same folio, so if any are + * dirty then the whole folio is dirty. This allows determining batch + * size without having to consider the dirty bit. Further, write protect + * it both in the parent and the child so that a future write will cause + * a CoW. As as an optimization, skip the wrprotect if all the ptes in + * the batch are already readonly. */ - if (vm_flags & VM_SHARED) + if (vm_flags & VM_SHARED) { pte = pte_mkclean(pte); - pte = pte_mkold(pte); + if (nr_writable > 0 && nr_writable < nr) + pte = pte_wrprotect(pte); + } else { + if (nr_dirty) + pte = pte_mkdirty(pte); + if (nr_writable) { + ptep_set_wrprotects(src_mm, addr, src_pte, nr); + pte = pte_wrprotect(pte); + } + } - if (!userfaultfd_wp(dst_vma)) + pte = pte_mkold(pte); + pte = pte_clear_soft_dirty(pte); + if (!enforce_uffd_wp) pte = pte_clear_uffd_wp(pte); - set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte); - return 0; + set_ptes(dst_vma->vm_mm, addr, dst_pte, pte, nr); + return nr; } static inline struct folio *page_copy_prealloc(struct mm_struct *src_mm, @@ -1021,6 +1115,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, int rss[NR_MM_COUNTERS]; swp_entry_t entry = (swp_entry_t){0}; struct folio *prealloc = NULL; + int nr_ptes; again: progress = 0; @@ -1051,6 +1146,8 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, arch_enter_lazy_mmu_mode(); do { + nr_ptes = 1; + /* * We are holding two locks at this point - either of them * could generate latencies in another task on another CPU. @@ -1086,16 +1183,21 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, * the now present pte. */ WARN_ON_ONCE(ret != -ENOENT); + ret = 0; } - /* copy_present_pte() will clear `*prealloc' if consumed */ - ret = copy_present_pte(dst_vma, src_vma, dst_pte, src_pte, - addr, rss, &prealloc); + /* copy_present_ptes() will clear `*prealloc' if consumed */ + nr_ptes = copy_present_ptes(dst_vma, src_vma, dst_pte, src_pte, + addr, end, rss, &prealloc); + /* * If we need a pre-allocated page for this pte, drop the * locks, allocate, and try again. */ - if (unlikely(ret == -EAGAIN)) + if (unlikely(nr_ptes == -EAGAIN)) { + ret = -EAGAIN; break; + } + if (unlikely(prealloc)) { /* * pre-alloc page cannot be reused by next time so as @@ -1106,8 +1208,9 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, folio_put(prealloc); prealloc = NULL; } - progress += 8; - } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end); + progress += 8 * nr_ptes; + } while (dst_pte += nr_ptes, src_pte += nr_ptes, + addr += PAGE_SIZE * nr_ptes, addr != end); arch_leave_lazy_mmu_mode(); pte_unmap_unlock(orig_src_pte, src_ptl); From patchwork Mon Dec 4 10:54:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478186 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10CCAC4167B for ; Mon, 4 Dec 2023 10:55:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 979086B029D; Mon, 4 Dec 2023 05:55:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 928FF6B029F; Mon, 4 Dec 2023 05:55:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7CA3C6B02A0; Mon, 4 Dec 2023 05:55:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6E3C06B029D for ; Mon, 4 Dec 2023 05:55:06 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3A454A08CD for ; Mon, 4 Dec 2023 10:55:06 +0000 (UTC) X-FDA: 81528828612.19.9415CA6 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf04.hostedemail.com (Postfix) with ESMTP id 90E1F40004 for ; Mon, 4 Dec 2023 10:55:04 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687304; a=rsa-sha256; cv=none; b=TL9M3BLkcSNvWVWdlXYULYq2dAFwzMkkseskbdO8+TClSt15uIWnS2rOuo2ejWDkcilAAN b0ekVrOjST8GfbSCVMhFuBcHje53kzq07h8vA9QVil4YSwhfa9hqYe8Muo9opFbRv7T0Cu HtSH/0e+QRvrx1glJCcPBbbZ0zVW+EQ= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; spf=pass (imf04.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687304; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EQA7si1ROP03u6D9py244k+Ey/Ri2m2/5MH2b5lgCqc=; b=niu7ANwd9EVMydts9Euu2RFSPZ0iSX1B+M26/0l/8O5NAHKlhd12iA2ba1jIFJ1gsgZFem tiz3iXy1aB0MqgDV2gp8G1T2lhPX/6vbJSKBMrOV1mE+LV3iZUUFMhM1fydpymit2szHTG 5bJM2n8Kt9IqBdJy8/f3cgRqFQ5wudU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1DD87169C; Mon, 4 Dec 2023 02:55:51 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6A2433F6C4; Mon, 4 Dec 2023 02:55:00 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 02/15] mm: Batch-clear PTE ranges during zap_pte_range() Date: Mon, 4 Dec 2023 10:54:27 +0000 Message-Id: <20231204105440.61448-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 90E1F40004 X-Stat-Signature: sq4ido6hp5x48tppkwzoj3z6uendmqpu X-HE-Tag: 1701687304-704169 X-HE-Meta: U2FsdGVkX1+wcXV9zl0j5zAHCM1DrmRZmqS7MLNm8FFqHvOsXwsTyc7DEZbBFP8WLF9bIsA9NAsD0huCVDZVTrmKb+7TIemlWUubjKZLi2cSuXaMuiMttoL/2OZ1DA3dOBxSjrifKgYxTOM/Sxduj5RQhJwgZ/mjQKJdzIzGS2yW6initjkZ1nQnCUtgcKgY3pi9T5bfDDI8KcTZ7QJy6rQ4c51tuMaSLsJMVVfSsixGKZXL7OewQPaVLOIHvqQOgQ5B8J/58HrIGoYt6zroKWbu9w01PcfKPtK5CAMq3L+eMcTw5X2pw/52OgU1bhWJgm5v0wDfFEFwqYPidHwSC/hhxlJwJ+pu+hLi657DDKRiW4MNIQVrqqd00AlLMOV2eDpTgFW2pLPxTFSlTV3stWGz+Vid/JcYAwrnrMLGkfTqQMVgDGc3cbyl+e1hkGIsTdXochYPgJ+6KFMq4RsXzZ6Ba2k3xQYqzllfMAv4eyS9a/T830mMrcEZEdV+GJ1glnVJa5qkgeJevp5datiR4TDSTSvNDgs79ywQ0HaCYz6DsKxCVsc9htVHyY033Jn6cq9TvxLz2B+QRKTbbTNHC2rKQ7IfHiPtrl+9BCDtac/e719RWVMmfWzRvBeg6wFDEAb3mOzwqvg9dovGT7D42fUfRotkSDkT1UuRto0/lY+1IjMidxGG0rKyV+5+gtrAuPonUzlte1us9t2ZquCIKe6PHd5dabyfoh9ZuTG7i4C+aEOKIwwInUWAtDGxsZSbFiC3uLXfd1+oNAYIF4hefvQISaWPb7X0gndYYmRSjnydaBXEWbFsdlacJ4SJVDpHk1LUm9w5CgECK0GTa0Y8bHU9wMd/UgqC7ExY2JHflQVhwti3MqE6LmSPAW4fIO46Li8kM+jg0ScjCs4hQAHfDYaBXGhxwX2FhMgAgulgfmZq0U1JaP6DXSi4sDSrlulM1j3pft/cG490TEvowwz vtvIwLf6 JZfQ5BL3RE/4ti809AcIT6lbVU2GhtalBCTp3xjXh5s3ji8T98EI/6fR5vK0c/VvUpYoNjbkHK17qu/O4NFpnANJ3Sy7PX4tJ30zvvXcWhRc52//xnXoDMSAvK8YOkNaVIQjhcqhot93bd+avY7gc+XoCygMYjdN2DrMW/W6Q47fwUnH/lLVuucpZG26qGpPVXemQJnf42XhcmGG2BGjZBsFGzPchTaQliiram8PEfB1xW/jzUgPEsHRgwwbtEBeg1SLt8YTAf9LWyxhsM3ir5h55RA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Convert zap_pte_range() to clear a set of ptes in a batch. A given batch maps a physically contiguous block of memory, all belonging to the same folio. This will likely improve performance by a tiny amount due to removing duplicate calls to mark the folio dirty and accessed. And also provides us with a future opportunity to batch the rmap removal. However, the primary motivation for this change is to reduce the number of tlb maintenance operations that the arm64 backend has to perform during exit and other syscalls that cause zap_pte_range() (e.g. munmap, madvise(DONTNEED), etc.), as it is about to add transparent support for the "contiguous bit" in its ptes. By clearing ptes using the new clear_ptes() API, the backend doesn't have to perform an expensive unfold operation when a PTE being cleared is part of a contpte block. Instead it can just clear the whole block immediately. This change addresses the core-mm refactoring only, and introduces clear_ptes() with a default implementation that calls ptep_get_and_clear_full() for each pte in the range. Note that this API returns the pte at the beginning of the batch, but with the dirty and young bits set if ANY of the ptes in the cleared batch had those bits set; this information is applied to the folio by the core-mm. Given the batch is garranteed to cover only a single folio, collapsing this state does not lose any useful information. A separate change will implement clear_ptes() in the arm64 backend to realize the performance improvement as part of the work to enable contpte mappings. Signed-off-by: Ryan Roberts --- include/asm-generic/tlb.h | 9 ++++++ include/linux/pgtable.h | 26 ++++++++++++++++ mm/memory.c | 63 ++++++++++++++++++++++++++------------- mm/mmu_gather.c | 14 +++++++++ 4 files changed, 92 insertions(+), 20 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 129a3a759976..b84ba3aa1f6e 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -75,6 +75,9 @@ * boolean indicating if the queue is (now) full and a call to * tlb_flush_mmu() is required. * + * tlb_get_guaranteed_space() returns the minimum garanteed number of pages + * that can be queued without overflow. + * * tlb_remove_page() and tlb_remove_page_size() imply the call to * tlb_flush_mmu() when required and has no return value. * @@ -263,6 +266,7 @@ struct mmu_gather_batch { extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct encoded_page *page, int page_size); +extern unsigned int tlb_get_guaranteed_space(struct mmu_gather *tlb); #ifdef CONFIG_SMP /* @@ -273,6 +277,11 @@ extern bool __tlb_remove_page_size(struct mmu_gather *tlb, extern void tlb_flush_rmaps(struct mmu_gather *tlb, struct vm_area_struct *vma); #endif +#else +static inline unsigned int tlb_get_guaranteed_space(struct mmu_gather *tlb) +{ + return 1; +} #endif /* diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 1c50f8a0fdde..e998080eb7ae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -635,6 +635,32 @@ static inline void ptep_set_wrprotects(struct mm_struct *mm, } #endif +#ifndef clear_ptes +struct mm_struct; +static inline pte_t clear_ptes(struct mm_struct *mm, + unsigned long address, pte_t *ptep, + int full, unsigned int nr) +{ + unsigned int i; + pte_t pte; + pte_t orig_pte = ptep_get_and_clear_full(mm, address, ptep, full); + + for (i = 1; i < nr; i++) { + address += PAGE_SIZE; + ptep++; + pte = ptep_get_and_clear_full(mm, address, ptep, full); + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} +#endif + /* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings diff --git a/mm/memory.c b/mm/memory.c index 8a87a488950c..60f030700a3f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1515,6 +1515,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, pte_t *start_pte; pte_t *pte; swp_entry_t entry; + int nr; tlb_change_page_size(tlb, PAGE_SIZE); init_rss_vec(rss); @@ -1527,6 +1528,9 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, do { pte_t ptent = ptep_get(pte); struct page *page; + int i; + + nr = 1; if (pte_none(ptent)) continue; @@ -1535,45 +1539,64 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, break; if (pte_present(ptent)) { - unsigned int delay_rmap; + unsigned int delay_rmap = 0; + bool tlb_full = false; + struct folio *folio = NULL; page = vm_normal_page(vma, addr, ptent); if (unlikely(!should_zap_page(details, page))) continue; - ptent = ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); + + if (likely(page)) { + folio = page_folio(page); + nr = folio_nr_pages_cont_mapped(folio, page, + pte, addr, end, + ptent, true, &i, &i); + nr = min_t(int, nr, tlb_get_guaranteed_space(tlb)); + } + + ptent = clear_ptes(mm, addr, pte, tlb->fullmm, nr); arch_check_zapped_pte(vma, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); - zap_install_uffd_wp_if_needed(vma, addr, pte, details, - ptent); + + for (i = 0; i < nr; i++) { + unsigned long subaddr = addr + PAGE_SIZE * i; + + tlb_remove_tlb_entry(tlb, &pte[i], subaddr); + zap_install_uffd_wp_if_needed(vma, subaddr, + &pte[i], details, ptent); + } if (unlikely(!page)) { ksm_might_unmap_zero_page(mm, ptent); continue; } - delay_rmap = 0; - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { if (pte_dirty(ptent)) { - set_page_dirty(page); + folio_mark_dirty(folio); if (tlb_delay_rmap(tlb)) { delay_rmap = 1; force_flush = 1; } } if (pte_young(ptent) && likely(vma_has_recency(vma))) - mark_page_accessed(page); + folio_mark_accessed(folio); } - rss[mm_counter(page)]--; - if (!delay_rmap) { - page_remove_rmap(page, vma, false); - if (unlikely(page_mapcount(page) < 0)) - print_bad_pte(vma, addr, ptent, page); + for (i = 0; i < nr; i++, page++) { + rss[mm_counter(page)]--; + if (!delay_rmap) { + page_remove_rmap(page, vma, false); + if (unlikely(page_mapcount(page) < 0)) + print_bad_pte(vma, addr, ptent, page); + } + if (unlikely(__tlb_remove_page(tlb, page, delay_rmap))) { + tlb_full = true; + force_flush = 1; + addr += PAGE_SIZE * (i + 1); + break; + } } - if (unlikely(__tlb_remove_page(tlb, page, delay_rmap))) { - force_flush = 1; - addr += PAGE_SIZE; + if (unlikely(tlb_full)) break; - } continue; } @@ -1624,7 +1647,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, } pte_clear_not_present_full(mm, addr, pte, tlb->fullmm); zap_install_uffd_wp_if_needed(vma, addr, pte, details, ptent); - } while (pte++, addr += PAGE_SIZE, addr != end); + } while (pte += nr, addr += PAGE_SIZE * nr, addr != end); add_mm_rss_vec(mm, rss); arch_leave_lazy_mmu_mode(); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 4f559f4ddd21..57b4d5f0dfa4 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -47,6 +47,20 @@ static bool tlb_next_batch(struct mmu_gather *tlb) return true; } +unsigned int tlb_get_guaranteed_space(struct mmu_gather *tlb) +{ + struct mmu_gather_batch *batch = tlb->active; + unsigned int nr_next = 0; + + /* Allocate next batch so we can guarrantee at least one batch. */ + if (tlb_next_batch(tlb)) { + tlb->active = batch; + nr_next = batch->next->max; + } + + return batch->max - batch->nr + nr_next; +} + #ifdef CONFIG_SMP static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_struct *vma) { From patchwork Mon Dec 4 10:54:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478187 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1B7BC4167B for ; Mon, 4 Dec 2023 10:55:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 360656B029F; Mon, 4 Dec 2023 05:55:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 30F0E6B02A1; Mon, 4 Dec 2023 05:55:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AF866B02A2; Mon, 4 Dec 2023 05:55:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0BD866B029F for ; Mon, 4 Dec 2023 05:55:10 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D2DEC1401A0 for ; Mon, 4 Dec 2023 10:55:09 +0000 (UTC) X-FDA: 81528828738.19.D50FCB2 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf04.hostedemail.com (Postfix) with ESMTP id 3A3894000E for ; Mon, 4 Dec 2023 10:55:08 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf04.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687308; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xsNAviUmR5F8SYwTUIrOFJJg1yOMZo9+rbtTxHdvUqw=; b=fYlOgVPwbCdyUc3yZhJDF9u1/ROniGe5Dyq7ZWzO+yMRKlvCRzwV4fPqDN7Y39CAuos8iW T2EVEKZp6K7ZEyng1pARmQoBjjKR9ijiIpwuomTOVzePuCvJyOzV9ZMn2lWclknn7PNNEO NTCbIWNglk5uyV8afj+4XetY89oi/28= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf04.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687308; a=rsa-sha256; cv=none; b=wFkdsSOaM2oC7Lv5RNVyaXm4vWNMw3UAnRUVedM/CHKOrJiuauJ+tTTkevY03kqm7OwYPG c4peCUWU2yre7kb50YK/bX9zODmGQKNx++DJmXLIl36wFSzAsiwTlD3MwqRpnM++/2VvpY c/H9dGXtDGYU9Jc+NPk9t4zm61yrzFM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B953C169E; Mon, 4 Dec 2023 02:55:54 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 119913F6C4; Mon, 4 Dec 2023 02:55:03 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 03/15] arm64/mm: set_pte(): New layer to manage contig bit Date: Mon, 4 Dec 2023 10:54:28 +0000 Message-Id: <20231204105440.61448-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 3A3894000E X-Stat-Signature: bxa8fehpemrp6yrm7qh8mtpp95hyrect X-Rspam-User: X-HE-Tag: 1701687308-178626 X-HE-Meta: U2FsdGVkX1+kCEHXZ3qrXn7ZaOdbq1J8sgxbeC+nD0Nm9yiHjb/kfsAozvykTOBF+8ma8nndydUT5um8zUnWrDxVoq4Xm/+b5xrlhK1iJDVzKMrQSpJCZCXTWQMVrgH3ZMUqMHH4mXrfx3rkiK9QZgylbKUjzw8yely3SmXQ/p8zknwTG2+xoyoBvww9BC+pnq796u5An4Vy2h5JDQR3VVemwMK553PgK9FiN4qXlsu1oRj9m84WsXcFxoZtxYsv+55QssHuDtpL7JzRspOh7yAyVKgEp0MPtFuEHCXGi0k1Qlz6TLn+uVPwGUDR50kuyo7FjUbxm0+8yTuiJkuqOVejVtMwtcuqKaXTQVacPQrPtmcZQcC3+yAkd3+94zAeCtmTDDqdA1i5aCuQNleSlGbrw/X8kUdaBtmtF98prB8i0jyxJWyWKbA2fkbPbb8DwxMaJy+d6TUZA/Q2rfnugUYYLt6tJkGO+07la4xbu3BUFKmWXGa9CXH32t6x4k1fkocQm1aG1ICo5ceDzv6cu9bv3qoMc0pjyqkJqVwqmq9PaDomu8nUZgnikyG0/7y6WPJHx3aRqHDiyeMWh/+J5pFz34srKKe608b0ErPfgoVq+8QnTYQ2nW5C0mxL7KtV+ed69NAbi2axLLT/CGPnNOL818oGsZDx4u8HxphM+jrNPMJ6my7O1+VNAwN4UDl0R2c0zFFXEuTEILAV2dBJQLoBSb/fyXjLjr2MdxbLm5bRrO1p+OzW3Z5+nmicTDnq49ou/6RzdeJJBbwJCSdA/vL5cg8eUTyVwB6zGTqielGNRc3lyq4jvEFvYjfqkzbQhVHuRvcle/gj731bwfRgyKabjUNgmh0E0SBPsAPKNQ/RfZ08V9EXYHkrwapKSKXgYn/GhyJLn2rW+CGSPve1yqPogWOCsIFgMfeSMHU41dVes8B6gbSwjCJdYbFGI/+rFAS176bHnwCuIwtX0jr LOT25enq osmGG7R/uyn999e+d0M2r2dBmecHjEDRR5Lsg7ORGx7K6hgkGVmioKXWmvW6S3LfsP+VU5H+ZIXpkGyqgQAHHOyDKk4wODFPHeKFq+/klOQoNCtFcC46AxfRORwhV0/nsuxlLJigy7msFoUxf4VF8Cmq74WYLmTuOGu9GzdUCQJ0gsd/SakNNawjFaaKqbs+P95tEgkXvQWkb0xkEeGO0wCkw1tOMVjWfZP7AqobHcSYkUYweKZPOk0x+Bc/DHEjWaVuTp7OXKDe554mXf6igh79QTQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 12 ++++++++---- arch/arm64/kernel/efi.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 2 +- arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 ++-- 7 files changed, 16 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b19a8aee684c..650d4f4bb6dc 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -93,7 +93,8 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm,addr,ptep) set_pte(ptep, __pte(0)) +#define pte_clear(mm, addr, ptep) \ + __set_pte(ptep, __pte(0)) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) /* @@ -261,7 +262,7 @@ static inline pte_t pte_mkdevmap(pte_t pte) return set_pte_bit(pte, __pgprot(PTE_DEVMAP | PTE_SPECIAL)); } -static inline void set_pte(pte_t *ptep, pte_t pte) +static inline void __set_pte(pte_t *ptep, pte_t pte) { WRITE_ONCE(*ptep, pte); @@ -350,7 +351,7 @@ static inline void set_ptes(struct mm_struct *mm, for (;;) { __check_safe_pte_update(mm, ptep, pte); - set_pte(ptep, pte); + __set_pte(ptep, pte); if (--nr == 0) break; ptep++; @@ -534,7 +535,7 @@ static inline void __set_pte_at(struct mm_struct *mm, { __sync_cache_and_tags(pte, nr); __check_safe_pte_update(mm, ptep, pte); - set_pte(ptep, pte); + __set_pte(ptep, pte); } static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, @@ -1118,6 +1119,9 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma, extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); + +#define set_pte __set_pte + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 0228001347be..44288a12fc6c 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -111,7 +111,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) pte = set_pte_bit(pte, __pgprot(PTE_PXN)); else if (system_supports_bti_kernel() && spd->has_bti) pte = set_pte_bit(pte, __pgprot(PTE_GP)); - set_pte(ptep, pte); + __set_pte(ptep, pte); return 0; } diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index c0a3301203bd..51cd4501816d 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -121,7 +121,7 @@ void __set_fixmap(enum fixed_addresses idx, ptep = fixmap_pte(addr); if (pgprot_val(flags)) { - set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); + __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); } else { pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr+PAGE_SIZE); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 555285ebd5af..5eade712e9e5 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -112,7 +112,7 @@ static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, if (!early) memset(__va(page_phys), KASAN_SHADOW_INIT, PAGE_SIZE); next = addr + PAGE_SIZE; - set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); + __set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); } while (ptep++, addr = next, addr != end && pte_none(READ_ONCE(*ptep))); } @@ -266,7 +266,7 @@ static void __init kasan_init_shadow(void) * so we should make sure that it maps the zero page read-only. */ for (i = 0; i < PTRS_PER_PTE; i++) - set_pte(&kasan_early_shadow_pte[i], + __set_pte(&kasan_early_shadow_pte[i], pfn_pte(sym_to_pfn(kasan_early_shadow_page), PAGE_KERNEL_RO)); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 15f6347d23b6..e884279b268e 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -178,7 +178,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, do { pte_t old_pte = READ_ONCE(*ptep); - set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); + __set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); /* * After the PTE entry has been populated once, we diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 8e2017ba5f1b..057097acf9e0 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -41,7 +41,7 @@ static int change_page_range(pte_t *ptep, unsigned long addr, void *data) pte = clear_pte_bit(pte, cdata->clear_mask); pte = set_pte_bit(pte, cdata->set_mask); - set_pte(ptep, pte); + __set_pte(ptep, pte); return 0; } diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 7b14df3c6477..230b607cf881 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -41,7 +41,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) * read only (code, rodata). Clear the RDONLY bit from * the temporary mappings we use during restore. */ - set_pte(dst_ptep, pte_mkwrite_novma(pte)); + __set_pte(dst_ptep, pte_mkwrite_novma(pte)); } else if ((debug_pagealloc_enabled() || is_kfence_address((void *)addr)) && !pte_none(pte)) { /* @@ -55,7 +55,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) */ BUG_ON(!pfn_valid(pte_pfn(pte))); - set_pte(dst_ptep, pte_mkpresent(pte_mkwrite_novma(pte))); + __set_pte(dst_ptep, pte_mkpresent(pte_mkwrite_novma(pte))); } } From patchwork Mon Dec 4 10:54:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478188 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B039C10F04 for ; Mon, 4 Dec 2023 10:55:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C602B6B02A1; Mon, 4 Dec 2023 05:55:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C0F516B02A3; Mon, 4 Dec 2023 05:55:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD75C6B02A4; Mon, 4 Dec 2023 05:55:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9DFE46B02A1 for ; Mon, 4 Dec 2023 05:55:13 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6E016C01B3 for ; Mon, 4 Dec 2023 10:55:13 +0000 (UTC) X-FDA: 81528828906.25.4827228 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id B5D40A0007 for ; Mon, 4 Dec 2023 10:55:11 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687311; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nX6cppQWykzFlGLv1/MzIfjs0FHZmU2vVGTO0jGCe+w=; b=jlXpEcKYKxvA1jP3agB50y0bsZVzyhR5qpM+BM0CPM3owsrU9xqS2mXY2deJC1WMBjAjkY dn8Vsi7NXXCLxf3VyF9zlkR5IRQGwoSaYnZJ0d1mdYu4pCufZqmzg7aH6SqEmYGwAHZxNr HcFi1EdyYHcEMl9M3n8xlL2hS64Z73g= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf15.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687311; a=rsa-sha256; cv=none; b=J86/TErgi4I7GsrrAUX2RdAXO5yjp4CIEbe6r9MogA+8OHHMxy/AMGxDgRazPsST46chSf GpxV5TLsHETS8SN5hs8KbgaZc2CdZY6KshnzgNzzfOv6u5gV3MgHqlEnHFphhjru2TgQ+2 BLj1LZKwNMnLYERHJZTE2LvdczhCbcM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 60A4E16A3; Mon, 4 Dec 2023 02:55:58 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id ACD7D3F6C4; Mon, 4 Dec 2023 02:55:07 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 04/15] arm64/mm: set_ptes()/set_pte_at(): New layer to manage contig bit Date: Mon, 4 Dec 2023 10:54:29 +0000 Message-Id: <20231204105440.61448-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B5D40A0007 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: xp1tp1hi1stw7etpb9y4pyzdjjpr65ho X-HE-Tag: 1701687311-602079 X-HE-Meta: U2FsdGVkX18owzZ0m5U+CNm8tw3y4e2tEh8CCzlUC/GJRowy/ErSpGnM1wA5Y/5Ov7ai53AOQ3KNyU1ELGjVwqTkpwTpuai3Q92AnmjSdoBWdEHEd0epFXYg6+IFCCDYuS6Y4swdwZHtlNnCXm/feE6NcI9o4TO7lw3p2i4fTk67VvbhRcN+OGqVI/gyWUVfVHSDjrlOMB6riymoCYQJaEXHkl0bO09Jk1BQg5aJzZDZquYg5TxJHHxoCEI8kRP4hLDBm8sAs1Ivc8ttHvn7U/uP+EbIRV5uh5Cw0C7c1B6x2OjizSEQmH1evad1iaMY+zKB10aFnC98UjYM/tibVI+7VTIgL/0wTCNzZEPAiIWTQ2r7M8XsA4rUKfHtz3ILpYEIj8vTkITogJpXAQNkmFr0rdQrQH8uKeQTuHCbGWDSgcqqHUw9r3PG33wJX2bBdqZDbDFjbvJtpkY+KPXNjrd2BD7KsM/zxRJ2l2z5IihaX3VFJl/DeZhpFNGikREk6cecpiMdkLdpxman94QJcj+BRNJmdRGgMb/RJSWKsCmBanxN8hZ+RVsVX+SCb+HsdKSUoX5fq5GKpBHokiQd20etys2ZWx9psN7T4O9ov3Iez137yx6LyFS85hEYyo/NEeDQShqBwtTM+lD5yrolXBNfNuIoPHkkFx6yZkVfCa9yHIqNzW8cN0Vid3AVBhjHK9AlRsDZVzlHQYw+mezYUrc7wojQOpd996WrR62xDKDtsJe5O2E/vpQ2GnG8VUjvtpo6QyUvuBaDwZmNeL6dRARdlR1AW5/8x+j1o2/EqqokTby+DTEqc5ICtOpWlHbGohfz224XtN37oWWXMrscJHlr8F5esSilBybHex09BQy+n9x224WGSXdu9oxzRwyK8FUEZfKjBYg6E3vHlRVbImU90dE8VaEmxM6FTUs3TPqTPgPDriadYJVFh/+7GSFjSrS+z2auzTTwHYIZ/LC 2M0zaWst EIbh9VR/Moyh735zTNeZkaW41fljSAYk6BlkenLa7dHbinkUMhBZAHgCt9H0z4+8KZS/ggS2IQ4yCht+hOH/f5MU+eOTeVRuMmymR5ETv61aZ0ahTyeR+xgurT6lRWTaWkC1oufM8Ibxn/hgxkv7nJMPOYljKe9Mqx6eTSmQmFZke+3CbNTDRTq5pyxaZHTd5XFaijwWSp18WAPsObaQK+q/5ZnsVl6qiypzvKJ1UvX61ZVIdgKW/YtawWLD5EZgbmLl4pIvngEr43NrCqlVzmeT83g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. set_pte_at() is a core macro that forwards to set_ptes() (with nr=1). Instead of creating a __set_pte_at() internal macro, convert all arch users to use set_ptes()/__set_ptes() directly, as appropriate. Callers in hugetlb may benefit from calling __set_ptes() once for their whole range rather than managing their own loop. This is left for future improvement. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 10 +++++----- arch/arm64/kernel/mte.c | 2 +- arch/arm64/kvm/guest.c | 2 +- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/hugetlbpage.c | 10 +++++----- 5 files changed, 13 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 650d4f4bb6dc..323ec91add60 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -342,9 +342,9 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) mte_sync_tags(pte, nr_pages); } -static inline void set_ptes(struct mm_struct *mm, - unsigned long __always_unused addr, - pte_t *ptep, pte_t pte, unsigned int nr) +static inline void __set_ptes(struct mm_struct *mm, + unsigned long __always_unused addr, + pte_t *ptep, pte_t pte, unsigned int nr) { page_table_check_ptes_set(mm, ptep, pte, nr); __sync_cache_and_tags(pte, nr); @@ -358,7 +358,6 @@ static inline void set_ptes(struct mm_struct *mm, pte_val(pte) += PAGE_SIZE; } } -#define set_ptes set_ptes /* * Huge pte definitions. @@ -1067,7 +1066,7 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) #endif /* CONFIG_ARM64_MTE */ /* - * On AArch64, the cache coherency is handled via the set_pte_at() function. + * On AArch64, the cache coherency is handled via the __set_ptes() function. */ static inline void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, @@ -1121,6 +1120,7 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, pte_t old_pte, pte_t new_pte); #define set_pte __set_pte +#define set_ptes __set_ptes #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index a41ef3213e1e..dcdcccd40891 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -67,7 +67,7 @@ int memcmp_pages(struct page *page1, struct page *page2) /* * If the page content is identical but at least one of the pages is * tagged, return non-zero to avoid KSM merging. If only one of the - * pages is tagged, set_pte_at() may zero or change the tags of the + * pages is tagged, __set_ptes() may zero or change the tags of the * other page via mte_sync_tags(). */ if (page_mte_tagged(page1) || page_mte_tagged(page2)) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index aaf1d4939739..629145fd3161 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -1072,7 +1072,7 @@ int kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm, } else { /* * Only locking to serialise with a concurrent - * set_pte_at() in the VMM but still overriding the + * __set_ptes() in the VMM but still overriding the * tags, hence ignoring the return value. */ try_page_mte_tagging(page); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 460d799e1296..a287c1dea871 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -205,7 +205,7 @@ static void show_pte(unsigned long addr) * * It needs to cope with hardware update of the accessed/dirty state by other * agents in the system and can safely skip the __sync_icache_dcache() call as, - * like set_pte_at(), the PTE is never changed from no-exec to exec here. + * like __set_ptes(), the PTE is never changed from no-exec to exec here. * * Returns whether or not the PTE actually changed. */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index f5aae342632c..741cb53672fd 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -254,12 +254,12 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - set_pte_at(mm, addr, ptep, pte); + __set_ptes(mm, addr, ptep, pte, 1); return; } if (!pte_cont(pte)) { - set_pte_at(mm, addr, ptep, pte); + __set_ptes(mm, addr, ptep, pte, 1); return; } @@ -270,7 +270,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, clear_flush(mm, addr, ptep, pgsize, ncontig); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); } pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, @@ -478,7 +478,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, hugeprot = pte_pgprot(pte); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); return 1; } @@ -507,7 +507,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pfn = pte_pfn(pte); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, From patchwork Mon Dec 4 10:54:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478189 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C509C4167B for ; Mon, 4 Dec 2023 10:55:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B74AD6B02A3; Mon, 4 Dec 2023 05:55:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B25546B02A5; Mon, 4 Dec 2023 05:55:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9ED706B02A6; Mon, 4 Dec 2023 05:55:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8CD726B02A3 for ; Mon, 4 Dec 2023 05:55:17 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 67B2EC01B2 for ; Mon, 4 Dec 2023 10:55:17 +0000 (UTC) X-FDA: 81528829074.27.0F4CF55 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id B431920027 for ; Mon, 4 Dec 2023 10:55:15 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687315; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YwPKQQiak6Wq4SeQDrdHtDBJcvIfgSIZ07B41IMN5A0=; b=4ERqG71lLYfwQK9CsW5U52Mjrnzy/kRp3DLQiWXoJlBfGcGsuqPLp5KCwAZlKR/MOqUYqB XGhY3mkJnuYwjSZZeT2oPgK2F4NYoH4tr4Fcr812+fvFXxvh1H4/xNPPg8CWDcTdef/Jof cp1U3nUpsNbiOI5mvbWbb6zvad8c4lc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687315; a=rsa-sha256; cv=none; b=UW5yh9G3OuD5yzlxvPcKY0lUppg4+5tjrPv8N1KlnR9FgqRGx0omVKpy4E3V2XkeVQRLyY FylYFJVqXiGDGHId3CFi4qIq5CbeK72CBDSemkmuGLfx0eq65xFCN2ff8E6SABk47wLB6y 3F+yEdaqXYaKpAXXdijoQR7Vrh7RrOA= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 07857139F; Mon, 4 Dec 2023 02:56:02 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 53EE93F6C4; Mon, 4 Dec 2023 02:55:11 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 05/15] arm64/mm: pte_clear(): New layer to manage contig bit Date: Mon, 4 Dec 2023 10:54:30 +0000 Message-Id: <20231204105440.61448-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B431920027 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: c4znddjdc8fwb6r6gqg3wgfmacb6m6ra X-HE-Tag: 1701687315-228622 X-HE-Meta: U2FsdGVkX1+pq1oa4YQc72WtTEhT45fPwGU/Yu0EoNGJaGmlO6uv8C2LWHMpkqG2NjpoRUGqupg4q/uGokdpGYmxfWCxp0E49S8bQcsQkJoELwtKDiLcWo9AlwQ956r/TpsujdjWxPjYu/iXiEpnyXLIyKXjUcxrYUny1KWXHZy77jDeH5ufZy0hllDpzIicswc0FvfDR06TF1RXxuraP9ItQQGEz/VDEYwnIVPGROJDOfo3yd37LnKDSu2Dm4tJVgrEEL51nRqHWA8XjVKVjU8Tc7yFFRXDBAmVBprzZWqlf1kvSWrGel3TGNKlrWvEy1tS1muGQTLTYAgOemYVgauLDqnXJMmE7LefGgpNuoNKwxa8h7jw2nToPmzcgw1nJ+FWXVxKn7TqWFY2fAMiNgjFj1ZZmLTfJQ5TZfddbQD+BcyHVw0DJF16Gf3kNiKNrkNFCxL+0hTCWB01a7nX2HFNzQXRshyUDBJoFB8+buXMCveGSejCDepqN1p5p7kjpkQgVY3+tgF+vxULO/q4pFH7i5zZcBsl3s9FhezlyygI6I28oqgK0CmK3vx/1K+azS2ncQTz87n/9O17uBUrwkXXG6pD3oYVgo2aTZVhuJA8cUgOF55t4sra+rN/qsIn2L3nco6PkrQWycullRrLR5fT3Mc1p63qyVX41BJipm8AWWmE0mb75nsJSClx36AiEPC6e1zhBFIFkEeNEIHrxz14FlVV5pcr0bYM5HAbJxSHuHrCDIDmY7RE9i5YQao74UDxLj0L+xUlK0f3VAOEwEl1dOQB2+elPaYB1RmkDytQcnSSlFxNDMrkkkg/ASGuTLztE01UyJ0qEC+i1oc9V0jbpygUbq/VDp0Oh+aBgqXpy6AwhMiMQZg80ck0hEqZSXDqlBdHXvwU+1eUbfJmfsMAuZgNNJILnbQp2NQLkWBH4KeGLIHEWfpUx2nFe0rjdw2JtK1TEUhYNEmdAGB MyzUtBx+ 4RcpdO0/fZFM9G0UiYZhOU2sS0Y1TQ/C9WYwMeXV6+Afk3Ye0474gAKxz8Jtd86xNK0ch44jsSVYsGcqDo5TO4FRRocQvadyeCOyfvqVpiwC1dnuzpzdPHaCTxYu4beoh45/lcpHfWqaFeKWuYDr+p8pF1pwbMFmIHJtzvHZ8VKV0//OiXVU48R4UsXoiJhoA/e7FnGRZPFLICZp9BQ0U3rIM8gEai20sqG5m3NHWp/afZaEOUs8PDmAYjEgaD59pQpzqskvgmyTIOcia1xsABWJDsQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 3 ++- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/arm64/mm/mmu.c | 2 +- 4 files changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 323ec91add60..1464e990580a 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -93,7 +93,7 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm, addr, ptep) \ +#define __pte_clear(mm, addr, ptep) \ __set_pte(ptep, __pte(0)) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) @@ -1121,6 +1121,7 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, #define set_pte __set_pte #define set_ptes __set_ptes +#define pte_clear __pte_clear #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 51cd4501816d..bfc02568805a 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -123,7 +123,7 @@ void __set_fixmap(enum fixed_addresses idx, if (pgprot_val(flags)) { __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); } else { - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr+PAGE_SIZE); } } diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 741cb53672fd..510b2d4b89a9 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -400,7 +400,7 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, ncontig = num_contig_ptes(sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - pte_clear(mm, addr, ptep); + __pte_clear(mm, addr, ptep); } pte_t huge_ptep_get_and_clear(struct mm_struct *mm, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index e884279b268e..080e9b50f595 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -859,7 +859,7 @@ static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, continue; WARN_ON(!pte_present(pte)); - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); if (free_mapped) free_hotplug_page_range(pte_page(pte), From patchwork Mon Dec 4 10:54:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90326C4167B for ; Mon, 4 Dec 2023 10:55:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C2B46B02A5; Mon, 4 Dec 2023 05:55:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 272EA6B02A7; Mon, 4 Dec 2023 05:55:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13AE36B02A8; Mon, 4 Dec 2023 05:55:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 034296B02A5 for ; Mon, 4 Dec 2023 05:55:21 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CF5CB1401A2 for ; Mon, 4 Dec 2023 10:55:20 +0000 (UTC) X-FDA: 81528829200.03.BD32CC8 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id 2661640015 for ; Mon, 4 Dec 2023 10:55:18 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687319; a=rsa-sha256; cv=none; b=gGpcOC9xRMVXKuJjI1UfqIhRluPJrLm/PKU9+f5e9wN+ZgHp4xc5z0NhC802FtD7XGdE/G +dHRHmbQK26idT7xa/V0AoT7JesYdxJVc1zrtFgtCbsMfu0JFV0r1Yi9Z6xcnN/xGisCm0 CMXXKiuB/YWah/+fFxhEkDSRM6FHfOs= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687319; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g7RKJubwZGuwzYxXs5rxmgWa6Yt6X+DHtCiVrTsfCrg=; b=R5PxrGSIqTgo1UhPDGQtWEP1p/dGkDQR5NmYzT4KT+kqoJY6hpsenOLWbz0xKZq68VLuXM 6aHUsyBkLDqIRG329z7i3IxaJhy5Is/JeGLOFBpLbeQf/3j/1bko71yhji4zfwBEzcv45l ie1baT+d0gG6aR0bjIZkrVKGIjkeYu0= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A2EA61650; Mon, 4 Dec 2023 02:56:05 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EF7523F6C4; Mon, 4 Dec 2023 02:55:14 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 06/15] arm64/mm: ptep_get_and_clear(): New layer to manage contig bit Date: Mon, 4 Dec 2023 10:54:31 +0000 Message-Id: <20231204105440.61448-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 2661640015 X-Stat-Signature: xmd9aifiqoubsk5z3k6ueixgxk59uict X-HE-Tag: 1701687318-752425 X-HE-Meta: U2FsdGVkX18psnixsOWlzPbdEDdvG6GxpNojjFKI8dYsK9QJ04/7xij3/0pcr8K+nCzt9U5+LY/C+ZJlyeroEidGJWnUVjGFMRyppdeWojdYj1IHUKMokVWYnJ/kQtjcmwfZ7BINPveDac61s7JxH3OuzyfSxEfa93TspFEbLSLnYWhneVBPjjFktcfDUlL0REeeS0PCN9rEQOFUGPfxcl7dvEKQhaVdkDkLxEoQ3BXGLHWkbwV0JkzMAXotgZ7sTGh7LkjHrFsU7qKJSZB55w6dHc9Az9qOrwgrUJvGVg4F/F9JNxAbRQCc10ljh0Qf84dJF29fdh05ek66dWRq/n4gUc0pnJS+lX0+x0OQe/R8f226OdUFCEFpQ9Am+cq8DXw937jPesGFcZiuGl6o3XpubP4WqpU1UEPs3iw5TBRPUU1/I7fSKaCm4nZAjviTlXkkhg6gCyJYwYp8vQZXnUTsDZhqoIK2SY9ZauKd7YBzf4zOdyDT+DVkju50CWPB8ko4mDIdL6UZo1jieQnGIo8zCktEEEXmusm9TAFS8EblIldvk8h+EiPy24++QGgnkVbWKKuSVwOM1pQ0Zn9oIKbcRfl4UaxlvNquC68iZW67shItunl15oXytTxhvfsk18iaek5vHNZ53Hu1YpdSe8vBRTsMtChcZ7PyQfhQ68agGMvzJZd4pvoWUKBKEf0gRZ+FIR+FDYs0eOKtgiTCigmxevcWAG54EPt+T0+prRyToeTkkQJzyoO/v4sVlDssmVJtBSu2r6z1c6RIS7Diw31uKK37TYYlQKNxRYeiAyY5XIjQy8k2V+Ln/+fG6AnjOzdh1AFL0P4dA0hPZ4+y+MsHJEoeJQrP1WREz1+wPuMAGTFqe/W75hPI89nJQFBOBqW1ojNMcMmuqz2JrBCuaJMLbg8/zce3YCsy/p60cvDCNLSG34zPynaygkr5aAN1yi5JYnyazY0TOZEu3j4 Rwlqxqn4 UdbXGArzHm+o/6tlS6+XFUufEX6tL8XaJORKX5luyJRxAMZy+ZJTSrzOQK5JKX4rHzzi5wOx/RZ5JE2zizZEeQ9Xxwa0+4ReDoknAdwROVrLKR9hOt9ApvfRSGopnF9c9ESjALfThDsvrMus1lMFD1/HPk4xBx/viJu2lkQi1NH/sXe3OxL+no9A9ON+txbuduRIlRb6kWy1vZknGkaaf1IGiLwukBHPvuFQPETcwiP5EgYWc0u+U/gGsboObDwQesE9+fqWb7pyoNfqt1AeZEuZ2yw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 5 +++-- arch/arm64/mm/hugetlbpage.c | 6 +++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1464e990580a..994597a0bb0f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -941,8 +941,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, +static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, unsigned long address, pte_t *ptep) { pte_t pte = __pte(xchg_relaxed(&pte_val(*ptep), 0)); @@ -1122,6 +1121,8 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, #define set_pte __set_pte #define set_ptes __set_ptes #define pte_clear __pte_clear +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +#define ptep_get_and_clear __ptep_get_and_clear #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 510b2d4b89a9..c2a753541d13 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -188,7 +188,7 @@ static pte_t get_clear_contig(struct mm_struct *mm, unsigned long i; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { - pte_t pte = ptep_get_and_clear(mm, addr, ptep); + pte_t pte = __ptep_get_and_clear(mm, addr, ptep); /* * If HW_AFDBM is enabled, then the HW could turn on @@ -236,7 +236,7 @@ static void clear_flush(struct mm_struct *mm, unsigned long i, saddr = addr; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - ptep_clear(mm, addr, ptep); + __ptep_get_and_clear(mm, addr, ptep); flush_tlb_range(&vma, saddr, addr); } @@ -411,7 +411,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, pte_t orig_pte = ptep_get(ptep); if (!pte_cont(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); + return __ptep_get_and_clear(mm, addr, ptep); ncontig = find_num_contig(mm, addr, ptep, &pgsize); From patchwork Mon Dec 4 10:54:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41E5FC4167B for ; Mon, 4 Dec 2023 10:55:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CFB5B6B02A7; Mon, 4 Dec 2023 05:55:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CABE36B02A9; Mon, 4 Dec 2023 05:55:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9B1C6B02AB; Mon, 4 Dec 2023 05:55:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AB9D06B02A7 for ; Mon, 4 Dec 2023 05:55:24 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8F8134018D for ; Mon, 4 Dec 2023 10:55:24 +0000 (UTC) X-FDA: 81528829368.07.58A5B2E Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf22.hostedemail.com (Postfix) with ESMTP id B8B0BC0020 for ; Mon, 4 Dec 2023 10:55:22 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687322; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ah49fTPYg1kjKkibdjx5mHLTxVRVE+SlIK3JQIq5PE4=; b=7HLmuONLy6ZIo4dQJJwCcrnlM0IJqDA8GI9dYJgQkWgSGgLkEbRLCZF+IJjvjdJNrGhwhZ qa53AQZuYTz0XsZSjPwhXID+IoTAE3LJceWChQ8ZfciPhcex4DdeqA2mxwR0k7Ol2ywDwX 97MEXLsIyaZQBWR6xhwok3a+eI67Cg0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf22.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687322; a=rsa-sha256; cv=none; b=VbC/b+tbnneXVd+f0XN1v62F+k82NFEhoo+9QeCiw3zDYDYcfL6jl5FYv7IilIB1FJZ91u I7x+OUmwPu/5Otu2cql8NwmtEIUb7INFd9D8VFSUyHGkm6c4UjJavVaocxA4okqb492o8Z Ev+mCO0nXtALYFxMKQlfYdTu1XFs6mw= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 49E5D1684; Mon, 4 Dec 2023 02:56:09 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 964FA3F6C4; Mon, 4 Dec 2023 02:55:18 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 07/15] arm64/mm: ptep_test_and_clear_young(): New layer to manage contig bit Date: Mon, 4 Dec 2023 10:54:32 +0000 Message-Id: <20231204105440.61448-8-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B8B0BC0020 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: twt1xg5i95wpq87udfa9nr5dz5t9issc X-HE-Tag: 1701687322-121201 X-HE-Meta: U2FsdGVkX1+zy33vha86i55yrDDSFPGES+mhuWexTniGyMN7/h4ILcItzpYlT7iiEq+hERACGiD33GHHQVzuD4Guk0uvZqWq+wMPjR4a5o3XVlh6zk37ceq+6u1xCIMF6nSfSYBCsQdXfG5764UNvBGoBw+rtb60Q0djEMFBDV4ssj7iJ88oz4teocI76Aado7+4Z/HTUgQig3xtkRSA5NFfLk25dWhVUMpJEFkY2QCNEtO22NDt92Kpli050//mJmowGQ5/QUc170NelvKV2kCy596lSB3eXzmvMWuozIe5Kcz8OucVC0u/NLdScYhtfQs8AsdSNAld7Bdoo4h7FWKq5qADaV/A+nOUTxXIvJCDQ+O38xZGzfwrnfiWgqKGhZiql/OQWsJSzQ/bj1WK2I9fziEibwi46yUNb0m9UWpNGmBKlfEEGRU/bVkwNKXO9neVX3RdkfRLwxZ8HhbxoxQ6stciAVAj4c3KrYa1+2PAtpERPY97poz4P49ns3dZg0c0KPZ0wTjjIvFLtjS1AP3FjdOqDZfN7C/ySwBStlwhiqGbu9m6yuq/oM/ef8Lz+hZFSEl1yVy+sbM8cVIiA/CFYp6/6R805vK4zdjVHPHbrQ343zDfmFnwsM8dQfBKALb7LF7tzHjco8FMhYGXPjhFNUvmVEuorKkPFnVbgokfVOmK4/pG477yZX7zbgzbgRTBD88f7Ho4k9BAoFQBEX5y7BAVaL3I8hgeXlL2mhNTYS7gqRb6ucStuqDpI+527XlcPykr0VSUz5725Dn1iACWj7NYBxNIRoodhKFDe2Vb8mv3NQLD/ya3vqRXcTfFB2fMAsIyVM8TiLaER2E2tVL5aNypRSaUkcAnJkKVn9r84g+29322P3w7/J27CWijIITxvs1si4o+WpeOhSaZ1mvb7TdyRtfG2RxBjbtAtOs11l4bEYy1WdgDmdO5AC3hTCptdiQBQ5OwWK4cu5s ONPLNG7r RiCM1Ond+Nk+n2w2UgWZofjQ4bK9MTmKxfdTp4/paHZEk1fvTEt4wAmTNnRf2cP8UDXBgKz2YHoYNAXGRgOfkrHQBAeVMJSXGVKA8gJ6RQyMQ/BAw5G3yxoUd+zBqjPD5Glps6YelBAYQf64INknri8hZEWpHo+6xtMXOOf2HBpkTbEOzBPRXCfKdewZkrXpgj09KRz/vIoabb7PSaVs6NLq5SoXEKQ6cP1w91EAL1dxIelylXpnLqQE++Ok7I3zvodoGq2pbgpds8vS1XojJvgxSrQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 994597a0bb0f..9b4a9909fd5b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -887,8 +887,9 @@ static inline bool pud_user_accessible_page(pud_t pud) /* * Atomic pte/pmd modifications. */ -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -static inline int __ptep_test_and_clear_young(pte_t *ptep) +static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) { pte_t old_pte, pte; @@ -903,18 +904,11 @@ static inline int __ptep_test_and_clear_young(pte_t *ptep) return pte_young(pte); } -static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long address, - pte_t *ptep) -{ - return __ptep_test_and_clear_young(ptep); -} - #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH static inline int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - int young = ptep_test_and_clear_young(vma, address, ptep); + int young = __ptep_test_and_clear_young(vma, address, ptep); if (young) { /* @@ -937,7 +931,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - return ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); + return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ @@ -1123,6 +1117,8 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, #define pte_clear __pte_clear #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG +#define ptep_test_and_clear_young __ptep_test_and_clear_young #endif /* !__ASSEMBLY__ */ From patchwork Mon Dec 4 10:54:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 882F9C4167B for ; Mon, 4 Dec 2023 10:55:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24D6C6B02A9; Mon, 4 Dec 2023 05:55:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 224DE6B02AC; Mon, 4 Dec 2023 05:55:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EC796B02AD; Mon, 4 Dec 2023 05:55:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F15256B02A9 for ; Mon, 4 Dec 2023 05:55:27 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CCD89A083F for ; Mon, 4 Dec 2023 10:55:27 +0000 (UTC) X-FDA: 81528829494.13.94FBA3C Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id 3CD201C000C for ; Mon, 4 Dec 2023 10:55:26 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687326; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eY6NiicJ90X6+1Q0eg6fxYOX0IQ4OXMtCPZ2hrvQ55c=; b=Z12SK2ABrWfQkUPLKGeb7DQH8FzENlr85uetRZnXEuNN+xoNSvPg8RdC0qqlsQrray5HxI PvoUhNsj+CzSTB0rQtDfF8H99zXkx5rXD1tBoiSptzzLPKCegCaXLgV2Nwx25ehjYqXVDW CSi/F28rdkRLW8/oahj4tickMMu88So= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf21.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687326; a=rsa-sha256; cv=none; b=zdPJb2/cBzkNxPRf3HHsd1PMXlIxkRS6kasdFaGHBOpBfjeAoiKKiI+iTQgOS09CMdG7x2 ny3OMz7LkFgrqZWf5/A46YvWxQkmte1Vr+qfOKKhDu/RJT5LEMozdRywNSs9g1AIg+ghBY vusxepk+qkpDrzg66eI+u/nh4C1wZYg= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E54BA169C; Mon, 4 Dec 2023 02:56:12 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3D9193F6C4; Mon, 4 Dec 2023 02:55:22 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 08/15] arm64/mm: ptep_clear_flush_young(): New layer to manage contig bit Date: Mon, 4 Dec 2023 10:54:33 +0000 Message-Id: <20231204105440.61448-9-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: pqkpcrssipdnqfuq7p6isbxfjtpwuj1m X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3CD201C000C X-HE-Tag: 1701687326-628806 X-HE-Meta: U2FsdGVkX19v5b7y0lxu5zk98LHaWE7dm285/wR+hgYy2mnuQZ1/raDbqXEjepCo9gZe/BLS03mKZBH623bvMDFOp0IqycnRP0AL1IurmBlbjWShyr/hoMf63bBBIRGRGW3zbMt1gLclOB6SuWsfe+qi4vFZWkDhUu5jaGKedoOas6e0hCxmZO0Hg91Jue0NwHv4axfLwlq5x7PVTurXiv8Y16su2y2x3TEOunNspd1Vdn2HhPfF2yUE+ZZQKqpIa/nYisUgdAWYBAVus4CPq52tIesKcy27g4MdgmmqghoLEzWr1WvPcm5VvUSOJ03bNAMZhDevOXifUO3nX7QQBMWwsJTcNXe0gw+Ti7737CG/SXIttdCDS/JR+bgeko1c2jtT4OLVI3Kvw63liCQjiQh5EWYqq6OAjjsvPDh4FK7cCxzd5/QwQJf/88wX3nEgpCSCB+6LmIYWYAFtEBz7fFrFb6Jdav7EfEz0Q35zytPMO3RO07z74lv3LT1U0Yu6u9UVzUlRxNDQNkJwPQk6T8rtNOULv2RFHkFcHzzu5vza3U2n8riAoKG1CLnKDs/rGBuP0xEg7Tx+z9riWdHkrKEa/Ima8FgsHpioSoN+n6Een88FiYgagMyClHwGOiTGI0eyGG7GHBjjiU/b8gckF4+iJmMgRb1JsHw7D/VT01Z46Xz+IKybr9xZ4g84bYe349IW5KgvW4ny7LU8MIpRAnJo5a7Vleny0AyQa4h6cjWfgfs+4GA30jlFxySr1nSeuN5XU5r1DSLFR296X5xJJpTRRhTeG0/HATS7Dz8EShrnkW4zqeQqVQrXfgRK3XGfm/XnM6HOwXF71sJksy3neB5Ov/CAXQnh1pY0sy3dFmfuP/yg7JkB2RsURXrdJFac+tzVbkv4xuXzsd9FFPyTjt0XxhHksCdjmbHaQELoWh/JvIA9xHYUD77J7h5vTtwLgSgB5ufLIUcZVkhadht pvTzKANE 0S2MJjIUGAkPBup6XXgjGXMhgmmD0tMRH4UzoILGzNrrNUNOzp+0PeAWxIPVwGY5/2XAq+E3k/rxLtPIUbwCdPFJl+L2yHTAUjeKp1pn18afeOoR4Q/Z4g7ESMkVjH/wlQ22lkZlzL3uxXRtFcF3rmWHKfSOPcNtL+RPxrYvkPakgOGvVdMGVwrzJuLPW0vReF/GxFpHsGAIYv/H9IrOt0jeeymEwyNHW/6bUHsFvfbnNQNH38y+89RrgfHlG0ylbH/10RQ5brgaTE3w1c+ges9FdmA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9b4a9909fd5b..fc1005222ee4 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -138,7 +138,7 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) * so that we don't erroneously return false for pages that have been * remapped as PROT_NONE but are yet to be flushed from the TLB. * Note that we can't make any assumptions based on the state of the access - * flag, since ptep_clear_flush_young() elides a DSB when invalidating the + * flag, since __ptep_clear_flush_young() elides a DSB when invalidating the * TLB. */ #define pte_accessible(mm, pte) \ @@ -904,8 +904,7 @@ static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, return pte_young(pte); } -#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, +static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { int young = __ptep_test_and_clear_young(vma, address, ptep); @@ -1119,6 +1118,8 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, #define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG #define ptep_test_and_clear_young __ptep_test_and_clear_young +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +#define ptep_clear_flush_young __ptep_clear_flush_young #endif /* !__ASSEMBLY__ */ From patchwork Mon Dec 4 10:54:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478193 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E77FC4167B for ; Mon, 4 Dec 2023 10:55:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 02C2D6B02AC; Mon, 4 Dec 2023 05:55:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F1D076B02AE; Mon, 4 Dec 2023 05:55:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DE4E66B02AF; Mon, 4 Dec 2023 05:55:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id CF2D86B02AC for ; Mon, 4 Dec 2023 05:55:31 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A6D70C01B3 for ; Mon, 4 Dec 2023 10:55:31 +0000 (UTC) X-FDA: 81528829662.17.D8BFE93 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf26.hostedemail.com (Postfix) with ESMTP id F03EB14001C for ; Mon, 4 Dec 2023 10:55:29 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf26.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687330; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2IQirIHWGTN0C5ucPumITjyI4YT5MYNWep5+T8WG/E4=; b=LHK6A9PSaFK2TIUKq77IpIb8Oax8AkHtjWy5ymCt2YNmSCXGSv3jZ19n0FAcq8W6g5xcAI 4i2UfRymtUyMX7/jPHarylvS1TmGUPvv1IrJgWbiaAGh8P1IJ3UySL4xeEPLXoGfuDmJun tBdLHAwes73zWdv09ki49xsosP9kryU= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf26.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687330; a=rsa-sha256; cv=none; b=8QKyJU++xEXS/omZhyqAaHGY1vzgNJQzsbr2meHFktkyOHbbXZhFay5nTCWu2PJEoRLx6G z+LVkEwRA2l8pBLzQhgEz72/ZJ3zyFY/x5G8omQGnGiT//JlbU/RcXNODfiiQwXyAAfY64 J256SPY/gp0poilHRfmrHLYeHKDVCCA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8C236169E; Mon, 4 Dec 2023 02:56:16 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D844F3F6C4; Mon, 4 Dec 2023 02:55:25 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 09/15] arm64/mm: ptep_set_wrprotect(): New layer to manage contig bit Date: Mon, 4 Dec 2023 10:54:34 +0000 Message-Id: <20231204105440.61448-10-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: F03EB14001C X-Stat-Signature: qtsobpexsjeumr5teeeb4f1yt1xgp9ts X-Rspam-User: X-HE-Tag: 1701687329-497280 X-HE-Meta: U2FsdGVkX1/PULnP2x0arf79hsISNrNUotMtAUREe5+MZYKLWZkWZlR/NhydVF2UJ9GlUB7bc9VtS4fJMoh4JVly+35EHXbIg68n0BHtFd5fYT6mBrrEelhjYUtMq+9XUnEvtODqQlMXtnFPWqREM0DaLlgCuE7ZJlrrK0R8dxzFC+vDQ98eaKhnHEN8L6nA693beYCM4SQsGlW6B/w1iNRrBxAAyJF9tcOrqlPiMhW+aAYoIXEe9EPtrT+7JP5Ei2LVS+gNZfn60GObJMqrAYXp9Ud7spCwY+/R0Gtf6xs2MeaOy3IIqfSLaswR2dYnxA5S7VrVS8uUA7+/SGMYKzL4x4Ff21kak920C49dm4VUKj9mtp9QgyZHdSb6Z9Qua26Ls6VItjyyWGNpYz2dECK14iPn0e+CSkh2+eAXDcxqObA31IdsBWVlwNVqYl0ReKJ5l8lince04h+HkLbtUXJ3ZHSnk3jjXL6u9tVxzOar6DIZo+MV9ScyC0x5irsb5mL6JKLmXWe4Qv3u61Eu3gCZdgi/8mx1EsPu7/wUVVElhQGIcgmOUDZ9+2frGuxkM1Kz5i+9Z17i3iXOUT7bhlPRQDEbbuoAk2YArhsNv8X6CHQDTNrPotGGqmeDHde/6Yt7hZlPKpHJfKUjRUWzpQLHlNrTn5Vfs9l2ehXygLMBqJKEcQC1CzCvLM3OJ1NCiN8G1IToGEupluZckz4THyQGsd5DQRiZNpGICm+wR72cy7DQ3m0l9Gn+0E4++y45++xtWckZsqK8qam9GT3WIoQFIfN0e2RuZah8recUyyPwRUaypgbzezUCL1/t4MWp3G+vTaWAhZndm+4ptJXF/6K60CG4raquiqMF058sBi7/nv5//FVKr6uG0ddvzsxG1gRqAA5rVPDO3YofkyeBfSFHIkslQOORBVo9R4MvbaLc1vzbD/YcdbHvH076t71aK1+Qogc2HXg1zgTxruT g7qkTRKI FO07pnnTWXUETOffCnuXp3jGRkH7uhpfDsqwLdHi/N0wQZiYz4eKiUO3UA/8lti3eoIaykD+aMbztLdSFQJC7PUcDB1e4X4rpEWgzlosSUsczlpk4WzZYVXZSMO6hl1NKVwzhg9jhb+i5y0NMrBiwW3j2JYl81uUH4gzi7SpZtOMInS2eORR6WzU2QORgPDTqwcALWjNnRuz8HIxyWHCDvR/ugxJdtgZgTwzrLM21ouofJw8ATNDlX035U1H7XQlbW/7nxMDu90tw1symK7/aJhhtjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 10 ++++++---- arch/arm64/mm/hugetlbpage.c | 2 +- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index fc1005222ee4..423cc32b2777 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -958,11 +958,11 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* - * ptep_set_wrprotect - mark read-only while trasferring potential hardware + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. */ -#define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) +static inline void __ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { pte_t old_pte, pte; @@ -980,7 +980,7 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - ptep_set_wrprotect(mm, address, (pte_t *)pmdp); + __ptep_set_wrprotect(mm, address, (pte_t *)pmdp); } #define pmdp_establish pmdp_establish @@ -1120,6 +1120,8 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, #define ptep_test_and_clear_young __ptep_test_and_clear_young #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +#define ptep_set_wrprotect __ptep_set_wrprotect #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index c2a753541d13..952462820d9d 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -493,7 +493,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t pte; if (!pte_cont(READ_ONCE(*ptep))) { - ptep_set_wrprotect(mm, addr, ptep); + __ptep_set_wrprotect(mm, addr, ptep); return; } From patchwork Mon Dec 4 10:54:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478194 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5837C4167B for ; Mon, 4 Dec 2023 10:55:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 77DE06B02AE; Mon, 4 Dec 2023 05:55:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 72DA46B02B0; Mon, 4 Dec 2023 05:55:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61DB06B02B1; Mon, 4 Dec 2023 05:55:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4FB326B02AE for ; Mon, 4 Dec 2023 05:55:35 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2D40F1C01E1 for ; Mon, 4 Dec 2023 10:55:35 +0000 (UTC) X-FDA: 81528829830.21.BDDB2F7 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id 8DB0714000A for ; Mon, 4 Dec 2023 10:55:33 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687333; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=H0BwGCXMY0Jm/jIsiVmBBtVJvXghyud81fHEkKj7GCo=; b=67uVcmK+gk7IwmzpNDRGRoHM7SlD/IAskY/xQj/szqB26mWhIZE7WxU2IsDv1JlxpdZbN4 0oTsMUrQnlVUySiT2JjL3bfDeKVCoZHEVHQ0QYmKNzGuHrJo5jIhPCfdPtiv8Pwadf4M5D x9k+ypo8+p9euwnUAt0C0ndNZAT/54A= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687333; a=rsa-sha256; cv=none; b=C91Z7RDp9z2GTAxVvKhPSKdmXsVr50hcPyHHdCVkUihGwmRN4sOrt5cZk+aYuCEpSXpyCH eKngiGKm4VDTNaHzb8hYuVAdP/xG00ArGoZ25jZYQ7IUhLbZ/y6fyW0WO/AGGkmgA9XNDq kBeAos86hJWjWiaIZSNG7fivSleHCaA= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3482816A3; Mon, 4 Dec 2023 02:56:20 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 7FF083F6C4; Mon, 4 Dec 2023 02:55:29 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 10/15] arm64/mm: ptep_set_access_flags(): New layer to manage contig bit Date: Mon, 4 Dec 2023 10:54:35 +0000 Message-Id: <20231204105440.61448-11-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8DB0714000A X-Rspam-User: X-Stat-Signature: 6jw19fmky68wzniydcyj8w9by5sr9ppd X-Rspamd-Server: rspam01 X-HE-Tag: 1701687333-248888 X-HE-Meta: U2FsdGVkX18w2zUmK4r62dmcwgR6XBXuMf7MmIYDj/hKr/pMLhIOLpTZ4f1clyjGIMf8EF5zdvpSY4ayQSuhXpX+6ytxAltMqYh5Mm1AAPkddcjIK7vTCAgrwEI+IVkluVjX8WAc6jhiFNI0/YgOLA3KASOrYf8ttLb9KM+ZXsKNxaNQHqA+mS+zcWDhjIM0XMoLIJ4QAJc3/fekX4/YBCT5Xd1+OAvA6kVWjjW7XCiauwGydx3Ijfkq4ooOQKWYUXl09oyq2BGTKs+pr4+IE8IxCN4eYq3cz2uLh6rdWe1K3tMX9GZvAIRiEO8PisRDhbuw1zMsrQup5lEBtCYr6224gvm2LUHcwoqFd0kOmU2Tvlw7Z2G5oGu9U6sXBzsAkcfXl05by8PVgR5g/nojA+/T1M1brGUUQXFbMW86eEP8w0iTC49kblsN9WwT0pGZQZQUyY3c8umLc4gB+s4f+uTR0W2C+ANhi0f7/8wsL6LU2wVoQIUfn2HJn5ddUVokVtOhzi6vuZgqp2N+FxYKOXWepTytse0T8wOp86fEFZzPkS8mSlMfctn9EVbmuwP4ySI0UHQytrUTglaubvkqL00E2sVb21LSYE2+ZTbsc6G9TPB6keoygV0USsVa4h9MDHDLu2zanOuU0eTy2JzdXn7hSfAhxUrqTOnI8gJ/N7caXuczczdSsGbm9UuNw7AJoGeHSu+HiF+TuCfEccWP7vYiykIopriTuBc7v0P0zKi8xFlrclgMWsHLTfp7lRiNlhPxFwangz4j9Pz8H0EGP1koLu4YgfjY4/6Ex/yVl9OqCKOojNqb6h9cROTUJsHkF39OCvj4JtX+60ygLal7L7dpQuEua5tgPNdy2JHp0eka63QYphUBMopz+ACF/7ovahUBjNqx7ZXNO/5saYpRsgZjNT4gnmw2Y9tqDo0LHuSaQ3c1Xr6knSWWdXjSvSvFrR8rLFk0OEmrk9tNX/H rYkIOo0I UAi+e3Pxo96nbZTD9FatritrcjcNsIXFfs+Giv1zO0a1FGAmI1C+hFFk5MjAmyhJCPLWGNwGBBQO7UdcZoTVul0A1lpYtMwG4JtHUnDaa+xZLfRGmagMQgIeHlvww78Am/v65sCTAfIvkGMrMEYczHM8yVXrSzSkATncxC6lJ3enr/8EpJ9oGf8U2Oc1RScV0BPc494x2opqn+IoImSQIkrq7pXDuJ0XRIuYt/NkQfMe3KSHKnt9RsOaAYoFnx173v28zWrXwgmgduJyii7na5xZpjw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 10 ++++++---- arch/arm64/mm/fault.c | 6 +++--- arch/arm64/mm/hugetlbpage.c | 2 +- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 423cc32b2777..85010c2d4dfa 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -312,7 +312,7 @@ static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, /* * Check for potential race with hardware updates of the pte - * (ptep_set_access_flags safely changes valid ptes without going + * (__ptep_set_access_flags safely changes valid ptes without going * through an invalid entry). */ VM_WARN_ONCE(!pte_young(pte), @@ -842,8 +842,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pte_pmd(pte_modify(pmd_pte(pmd), newprot)); } -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -extern int ptep_set_access_flags(struct vm_area_struct *vma, +extern int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); @@ -853,7 +852,8 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty) { - return ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); + return __ptep_set_access_flags(vma, address, (pte_t *)pmdp, + pmd_pte(entry), dirty); } static inline int pud_devmap(pud_t pud) @@ -1122,6 +1122,8 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_WRPROTECT #define ptep_set_wrprotect __ptep_set_wrprotect +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +#define ptep_set_access_flags __ptep_set_access_flags #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index a287c1dea871..7cebd9847aae 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -209,9 +209,9 @@ static void show_pte(unsigned long addr) * * Returns whether or not the PTE actually changed. */ -int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, - pte_t entry, int dirty) +int __ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty) { pteval_t old_pteval, pteval; pte_t pte = READ_ONCE(*ptep); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 952462820d9d..627a9717e98c 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -459,7 +459,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, pte_t orig_pte; if (!pte_cont(pte)) - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); + return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); ncontig = find_num_contig(mm, addr, ptep, &pgsize); dpfn = pgsize >> PAGE_SHIFT; From patchwork Mon Dec 4 10:54:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478195 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0F9EC10F04 for ; Mon, 4 Dec 2023 10:55:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 385A26B02B0; Mon, 4 Dec 2023 05:55:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 335CB6B02B2; Mon, 4 Dec 2023 05:55:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1B0606B02B3; Mon, 4 Dec 2023 05:55:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0BCA56B02B0 for ; Mon, 4 Dec 2023 05:55:39 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C8E7C120197 for ; Mon, 4 Dec 2023 10:55:38 +0000 (UTC) X-FDA: 81528829956.23.3EE37ED Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id 231041C0015 for ; Mon, 4 Dec 2023 10:55:36 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687337; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/VxtEV8E64tqDjSTd4uv2FQfTUg5XNfrhh1313bOMIY=; b=iZ4i04FeHi83sCZeOUMc6INKzpdvMlTHr8sjVI2Eskl/lc7A1DkZn9MgvWXF6tKKgTYfPy ovfQye1AzJsrv32z9WZ6OiCpaKLB7Lsh5imDDqSji2VlSXsM6qeSqRBXleC67yREyvBAgZ HyShu+qZRo/YN3PbFkhAurYqFvwW/3c= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687337; a=rsa-sha256; cv=none; b=20wefyRNCNL/mnaNqtyeJXFEZYyqma9oOiLmPdei06KrBFmvWPv3y4sj3sCIvTFm0GTi4t V+vEtwvFKRt8L/CI5kd5Klbg0eBbD4/flSuH3/fR5oslWUAxA/9J2pkDnfh7kmXpqAjCxu 6PRNsORI/p5PBLrmZSRaWAZhk2EP1Rw= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D0283169E; Mon, 4 Dec 2023 02:56:23 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 281033F6C4; Mon, 4 Dec 2023 02:55:33 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 11/15] arm64/mm: ptep_get(): New layer to manage contig bit Date: Mon, 4 Dec 2023 10:54:36 +0000 Message-Id: <20231204105440.61448-12-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 231041C0015 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: h3m9hfpenf89g7ni4nhw43sfjwy8dxyc X-HE-Tag: 1701687336-676832 X-HE-Meta: U2FsdGVkX1/V+eorC48UiwlY/jPWDsBdfX0q3OGXwQThoMxJRMjr+Vk4QDyJ5o0VFE/e6YH/ygVV/HFf9hMMMQXfZoEZdf2FA5hOCw86tUGBg5CICSuEmtLT0TxkA+cmyzh+FwF18gOA3HsgBbvhsm3+G0n9BPvXxNIsbgI90mUlWKa4leWhBNqA15eZH4HRii7svvdkyruTKHfkj1fB07hQnahBOvLMRWqUd50ZL+k42fnu4ClXh2wTlvUh21Ex6ygP3/NPCVT62gKW0POqWB86dMV50W3vWH+0tAvBkwEAgVxXWAtIkA7/w5LbsYBbH6qKAeEGnhAIXWhUpYr8D8ALmpKZthUqZ/Y8qcNTY6chy7duUaIsdbMTHAD3/WicjzL56N0Oq9UsFpnmCZq45nr4x8AkJzyOwWak6CuozPF4soRmGJ/P+YOD136WtwNUzWGaPT2Sp94WLz7zWcBJbl/KmZ43VwbCj0BBt1n6Rcm/EWiPH9qiw7V/3bywIiVhozrUEW3zd9dnYaysgO51+4901vMCATnFdOYSeJul0C3EzK4nDqJ6lCKa72BiFFmwxQBa2fRzGNcpZDzv7HMmbMw199CE1oJOcYo+9LmLi1YM81GbkvwXwDSBkkg4RpUeUxYsAfcZUiWLLl2JL81bb1XiW/CQbUqjFb5MfPeZdgvVrs4jDbGe64WtcGMg43XwVqLFMD/l0oJ3BZCUUZeTOdAZ0eEDE07wFv7mDjc3UlVI+Xs5IjCnZ2fSdr2km79AA2j8NETRHNQd9mDZkK45cOKl0PJBe/+WHMu5EaYTSJkoKcGLEH4+amn1U2Nunq9JhKjt5dtNQGqKAUwSkyr/HKU08LL2HGRN1TFKOGq09RTqkJVEW3VitLEn05Me1h0WHvWvmmah+l03naK0fYluI2cCeHNrUPigA0IUCO4d2WZ1RCpgsNgEPErfe73Wh2/y5v6LHgoJcJEQJteAc2y V+iuvHYd R3yUBkG/9S5+tS53dKy80NjqX9Xw3RBpWmoE08skJTWpz+LbRSAHc7qKiQzNzxUV945hf+ihZPAUWzJu8PsNfB9OogeaTUIESiN1NVDALv6IQySsIHcXgvvgkyuX0Qf3QJkRaDKBp3C3m87guggVKTThIt+dZeKZ2v1wnwzSqI2kd8a/4AAuak5nFkARBV+TADWZpRMgH45MMFdYuu88CviARRPqMXOQJgTguKRrT72Rp7bpzGxV648KpJPUTlwNYhxXA+c8EAjRVOz9nDjCKph4eXQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. arm64 did not previously define an arch-specific ptep_get(), so override the default version in the arch code, and also define the private __ptep_get() version. Currently they both do the same thing that the default version does (READ_ONCE()). Some arch users (hugetlb) were already using ptep_get() so convert those to the private API. While other callsites were doing direct READ_ONCE(), so convert those to use the appropriate (public/private) API too. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 12 +++++++++--- arch/arm64/kernel/efi.c | 2 +- arch/arm64/mm/fault.c | 4 ++-- arch/arm64/mm/hugetlbpage.c | 18 +++++++++--------- arch/arm64/mm/kasan_init.c | 2 +- arch/arm64/mm/mmu.c | 12 ++++++------ arch/arm64/mm/pageattr.c | 4 ++-- arch/arm64/mm/trans_pgd.c | 2 +- 8 files changed, 31 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 85010c2d4dfa..6930c14f062f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -276,6 +276,11 @@ static inline void __set_pte(pte_t *ptep, pte_t pte) } } +static inline pte_t __ptep_get(pte_t *ptep) +{ + return READ_ONCE(*ptep); +} + extern void __sync_icache_dcache(pte_t pteval); bool pgattr_change_is_safe(u64 old, u64 new); @@ -303,7 +308,7 @@ static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, if (!IS_ENABLED(CONFIG_DEBUG_VM)) return; - old_pte = READ_ONCE(*ptep); + old_pte = __ptep_get(ptep); if (!pte_valid(old_pte) || !pte_valid(pte)) return; @@ -893,7 +898,7 @@ static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, { pte_t old_pte, pte; - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_mkold(pte); @@ -966,7 +971,7 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, { pte_t old_pte, pte; - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_wrprotect(pte); @@ -1111,6 +1116,7 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); +#define ptep_get __ptep_get #define set_pte __set_pte #define set_ptes __set_ptes #define pte_clear __pte_clear diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 44288a12fc6c..9afcc690fe73 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -103,7 +103,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) { struct set_perm_data *spd = data; const efi_memory_desc_t *md = spd->md; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = __ptep_get(ptep); if (md->attribute & EFI_MEMORY_RO) pte = set_pte_bit(pte, __pgprot(PTE_RDONLY)); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 7cebd9847aae..d63f3a0a7251 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -191,7 +191,7 @@ static void show_pte(unsigned long addr) if (!ptep) break; - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); pr_cont(", pte=%016llx", pte_val(pte)); pte_unmap(ptep); } while(0); @@ -214,7 +214,7 @@ int __ptep_set_access_flags(struct vm_area_struct *vma, pte_t entry, int dirty) { pteval_t old_pteval, pteval; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = __ptep_get(ptep); if (pte_same(pte, entry)) return 0; diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 627a9717e98c..52fb767607e0 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -152,14 +152,14 @@ pte_t huge_ptep_get(pte_t *ptep) { int ncontig, i; size_t pgsize; - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); if (!pte_present(orig_pte) || !pte_cont(orig_pte)) return orig_pte; ncontig = num_contig_ptes(page_size(pte_page(orig_pte)), &pgsize); for (i = 0; i < ncontig; i++, ptep++) { - pte_t pte = ptep_get(ptep); + pte_t pte = __ptep_get(ptep); if (pte_dirty(pte)) orig_pte = pte_mkdirty(orig_pte); @@ -184,7 +184,7 @@ static pte_t get_clear_contig(struct mm_struct *mm, unsigned long pgsize, unsigned long ncontig) { - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); unsigned long i; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { @@ -408,7 +408,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, { int ncontig; size_t pgsize; - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); if (!pte_cont(orig_pte)) return __ptep_get_and_clear(mm, addr, ptep); @@ -431,11 +431,11 @@ static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig) { int i; - if (pte_write(pte) != pte_write(ptep_get(ptep))) + if (pte_write(pte) != pte_write(__ptep_get(ptep))) return 1; for (i = 0; i < ncontig; i++) { - pte_t orig_pte = ptep_get(ptep + i); + pte_t orig_pte = __ptep_get(ptep + i); if (pte_dirty(pte) != pte_dirty(orig_pte)) return 1; @@ -492,7 +492,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, size_t pgsize; pte_t pte; - if (!pte_cont(READ_ONCE(*ptep))) { + if (!pte_cont(__ptep_get(ptep))) { __ptep_set_wrprotect(mm, addr, ptep); return; } @@ -517,7 +517,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, size_t pgsize; int ncontig; - if (!pte_cont(READ_ONCE(*ptep))) + if (!pte_cont(__ptep_get(ptep))) return ptep_clear_flush(vma, addr, ptep); ncontig = find_num_contig(mm, addr, ptep, &pgsize); @@ -550,7 +550,7 @@ pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ - if (pte_user_exec(READ_ONCE(*ptep))) + if (pte_user_exec(__ptep_get(ptep))) return huge_ptep_clear_flush(vma, addr, ptep); } return huge_ptep_get_and_clear(vma->vm_mm, addr, ptep); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 5eade712e9e5..5274c317d775 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -113,7 +113,7 @@ static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, memset(__va(page_phys), KASAN_SHADOW_INIT, PAGE_SIZE); next = addr + PAGE_SIZE; __set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); - } while (ptep++, addr = next, addr != end && pte_none(READ_ONCE(*ptep))); + } while (ptep++, addr = next, addr != end && pte_none(__ptep_get(ptep))); } static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 080e9b50f595..784f1e312447 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -176,7 +176,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, ptep = pte_set_fixmap_offset(pmdp, addr); do { - pte_t old_pte = READ_ONCE(*ptep); + pte_t old_pte = __ptep_get(ptep); __set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); @@ -185,7 +185,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), - READ_ONCE(pte_val(*ptep)))); + pte_val(__ptep_get(ptep)))); phys += PAGE_SIZE; } while (ptep++, addr += PAGE_SIZE, addr != end); @@ -854,7 +854,7 @@ static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, do { ptep = pte_offset_kernel(pmdp, addr); - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); if (pte_none(pte)) continue; @@ -987,7 +987,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, do { ptep = pte_offset_kernel(pmdp, addr); - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); /* * This is just a sanity check here which verifies that @@ -1006,7 +1006,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, */ ptep = pte_offset_kernel(pmdp, 0UL); for (i = 0; i < PTRS_PER_PTE; i++) { - if (!pte_none(READ_ONCE(ptep[i]))) + if (!pte_none(__ptep_get(&ptep[i]))) return; } @@ -1475,7 +1475,7 @@ pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ - if (pte_user_exec(READ_ONCE(*ptep))) + if (pte_user_exec(ptep_get(ptep))) return ptep_clear_flush(vma, addr, ptep); } return ptep_get_and_clear(vma->vm_mm, addr, ptep); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 057097acf9e0..624b0b0982e3 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -36,7 +36,7 @@ bool can_set_direct_map(void) static int change_page_range(pte_t *ptep, unsigned long addr, void *data) { struct page_change_data *cdata = data; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = __ptep_get(ptep); pte = clear_pte_bit(pte, cdata->clear_mask); pte = set_pte_bit(pte, cdata->set_mask); @@ -246,5 +246,5 @@ bool kernel_page_present(struct page *page) return true; ptep = pte_offset_kernel(pmdp, addr); - return pte_valid(READ_ONCE(*ptep)); + return pte_valid(__ptep_get(ptep)); } diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 230b607cf881..5139a28130c0 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -33,7 +33,7 @@ static void *trans_alloc(struct trans_pgd_info *info) static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) { - pte_t pte = READ_ONCE(*src_ptep); + pte_t pte = __ptep_get(src_ptep); if (pte_valid(pte)) { /* From patchwork Mon Dec 4 10:54:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48C98C4167B for ; Mon, 4 Dec 2023 10:55:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D4B476B02B2; Mon, 4 Dec 2023 05:55:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D21BA6B02B4; Mon, 4 Dec 2023 05:55:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C10B36B02B5; Mon, 4 Dec 2023 05:55:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B10D66B02B2 for ; Mon, 4 Dec 2023 05:55:42 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 86D71C0191 for ; Mon, 4 Dec 2023 10:55:42 +0000 (UTC) X-FDA: 81528830124.18.5296BCA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf27.hostedemail.com (Postfix) with ESMTP id F3E6440013 for ; Mon, 4 Dec 2023 10:55:40 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf27.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687341; a=rsa-sha256; cv=none; b=Zh0HU4Buk5Ktge7xXDN7hEX/IEXCfnu4DXVKTNZzH40xh3+TFUl8nVyE2isLdT5pJPzcvA RFzfQ2bwQu0j5Q88PPWCTZtnq5lcZOc77JmX5KC/157caOI8YogCvP4siSwQPY/u3F2t0q fU+TmCC11biPgKQr6Zudv1Zz2K41TKU= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf27.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687341; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mOqMiM5NGpNjoYnvzB47gVofKLzjpD8ExtOVjQRbqJI=; b=ukKHLjXwXT9WQ311xE0HqwyRgqfwefANbLZKEbG0NkKEYQ/ynaKZNFryybkEE3BDTrFiZR XhC75WMHeE2KRNuk6NJPctGgwlmyI7s1W42tKeYSyOY8DAthPPm5+njM5aFwuBmGy8i2tk x7bbu/7ty5ZMs6yGS4Z/77TvUDSoZFI= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7825816F2; Mon, 4 Dec 2023 02:56:27 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C3DCA3F6C4; Mon, 4 Dec 2023 02:55:36 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 12/15] arm64/mm: Split __flush_tlb_range() to elide trailing DSB Date: Mon, 4 Dec 2023 10:54:37 +0000 Message-Id: <20231204105440.61448-13-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: F3E6440013 X-Stat-Signature: j5hm8tgtqx9cmrtznid7h19nuctx8mza X-Rspam-User: X-HE-Tag: 1701687340-373254 X-HE-Meta: U2FsdGVkX1+ZUxf4w5upf7QuP+7MVASaY53SfPnGo8HTsgxJMO4Hy2bmtXLmclsBt2cB5oam7chWwpCxulGb/JXjwsQKH1pS7ZdWvasN9mnlaUmJrqVufi4HQnFIbhnSvpxm1Zrwlr1q+SGz8G1LGpWWZcIAXXbI15eMHyTFjCbyFEf0DJ12oww9+aqB3QX5HIs5bWmlIn17k9XoGjAw2B19KIQsUmI1rK9xT8j7pHxuhk/7BsResROUTHx1K0lZxKwemSixMdVOutyk+BnEAuGQTCftcHr17vHoGXnl8ZDHSOnX0UB6rqcZzk2/Fv7sWPClX00IGwXhbkSxOBZqCFEesjT7tv6prAIt0v5gG3g8X1rhHChOvuzD/R+KRJh/0nC7/7u1aC8HM3weQiA2r3Am5hQtk+RGbSXGtW1XcNRivRgLJfgEbGhs9fz2vZ99ZnJk/RmZoDq9XJOc5xHUDWktNh9vv7GDQn+rMGH0L6qOf5gA5dQ9MUAzBdoXIr0VjwcMlwVhejeS81u8YAhGE5v404j7zKtVIjBhJpHAolzgph5GlsIIkMTsqIFv5rrYHOXRgGk6way+muEJ++oZsrB3ZDEN8LRIEaVhOYJoIN4HnGaGZbC3oWhiD+dkkV95YrDIS/HRv/MmMlFnBEu3+pXt65i2srPGr0q47o3MugHFiB/CA4w/ZNFEzlhwrJTiEDMOJ6NMnL/Ntzk1S5X9yJQktS4t22156AEq/McRkhQr9p5EUtb86qu6eIIyqTuSeqfWorke7dB1OTOWfe8gzBMCOBt83I5ZH89Qh1jduagHTFbS0cDmvlkaUMzUEV4oxKUwHlc4EBQv8pX70sGrqg+ABn6n2edfOFbjnaBuGOm2YAbkx+XI8ooA8JpCkluvc9zKSmc7UW1/8LHbVbJbjRvAMEHs5DELU0gtl3DSNXndtdMNWD1qpsiGf3amBJLrKxgMGXBV6irbWeYrqI2 tQCPrY+G l/qoO+HoWpvP7Auc3tbc/IJnHAGJi8Nd6GAamAFZSkUuVu6HF72Rbof7W6hZLuspyFuZN+0nVov4kfKRy1Bb07XCVlXh3VQGLImDY0v3KnmjYIIdvm9j0IyjHsIvkyYa+4f75xrcVOaaAPuO0irHLV6n2VbJayfpSdqln9H4B9RyRCBPPnzCaZHpyKSFM9berfPehHyHFRllwVWDPo9/PB3krx36SqjC9KggwPeDNdEA6lMnfqD/dBBCIv6sIyWprUqPFVFqemQYVNFQLS5SwHTCLjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Split __flush_tlb_range() into __flush_tlb_range_nosync() + __flush_tlb_range(), in the same way as the existing flush_tlb_page() arrangement. This allows calling __flush_tlb_range_nosync() to elide the trailing DSB. Forthcoming "contpte" code will take advantage of this when clearing the young bit from a contiguous range of ptes. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/tlbflush.h | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index bb2c2833a987..925ef3bdf9ed 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -399,7 +399,7 @@ do { \ #define __flush_s2_tlb_range_op(op, start, pages, stride, tlb_level) \ __flush_tlb_range_op(op, start, pages, stride, 0, tlb_level, false) -static inline void __flush_tlb_range(struct vm_area_struct *vma, +static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) @@ -431,10 +431,19 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, else __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true); - dsb(ish); mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, start, end); } +static inline void __flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end, + unsigned long stride, bool last_level, + int tlb_level) +{ + __flush_tlb_range_nosync(vma, start, end, stride, + last_level, tlb_level); + dsb(ish); +} + static inline void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { From patchwork Mon Dec 4 10:54:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CAA9C10DC1 for ; Mon, 4 Dec 2023 10:55:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0EE016B02B4; Mon, 4 Dec 2023 05:55:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 09DDE6B02B6; Mon, 4 Dec 2023 05:55:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA8696B02B7; Mon, 4 Dec 2023 05:55:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DAD1F6B02B4 for ; Mon, 4 Dec 2023 05:55:46 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6B23BA08BF for ; Mon, 4 Dec 2023 10:55:46 +0000 (UTC) X-FDA: 81528830292.19.9D9E4DB Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id AB8DE100002 for ; Mon, 4 Dec 2023 10:55:44 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687344; a=rsa-sha256; cv=none; b=77Gj1AQ2ntbnbEWWN03iqL3zMVXy8QYlQMxt3ISm2NEEN1p6WvtuBLsFRiiYc5wb4DJJ6X AUvwjybFnYpkWaRC2MokHdpHvgl/s13s3M7vwMfrbtU+RSfxYve9wwAv7mp5qJUyFHuyun 4wpyFziekxsDiJAnZr0sBYgz/OcoKi0= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687344; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UbVQkm8T6rKAC0yu2lDD8CMrEkvawWpebEWEcG1EutA=; b=8QB8lBpouoP0WbrLFMU/0iS8DgkTxvhalo/MyBqI5slC8ARNY2vAfup1rqTMIhZLkXgJtx LMHfgxIfFHYgiHRWDlLbTNxfN8WldDuCsrzlsCzHnvwrZdRD20OUTramONUff5pHjFEO+v uT7Fa0h7QW2LAijR4slddbdTHAYW9zM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3B54F16F3; Mon, 4 Dec 2023 02:56:31 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6C52D3F6C4; Mon, 4 Dec 2023 02:55:40 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 13/15] arm64/mm: Wire up PTE_CONT for user mappings Date: Mon, 4 Dec 2023 10:54:38 +0000 Message-Id: <20231204105440.61448-14-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: AB8DE100002 X-Stat-Signature: kq7nnq17u4njfdwwhpj3hjw715d1aajn X-Rspam-User: X-HE-Tag: 1701687344-750776 X-HE-Meta: U2FsdGVkX1+w4VypWmyj8oupZnSkBCzFSYrdIk2HESbrAVsTqEahmQZoX3kdbCntyEWtSXZcQFqkbFJjXe9GlylIP6VI3o1DyrowQoZgijs9ZxXt95OrQAxCfFAU0tRAbG89dVC4rZ1mQYE0txj3Pi1hxo4eXTD9E31kcnNwEuThFAIeO8vYDMxavZOYcDb3+sntGPvdr/wusfssjdECx3WC0V6DoqmPyR3oSpw50+Uvcf9J1yEZXI6sKceARoA5vhKaA1IJ+lBI3p8J1PawLT7IOhhc2rqL8sN/DjARQljc0yHXHGWPrqjmCm697QhGum//6kSNj+Z5zCYSm2IUG3MJBPJrq4ybu+s62+pRafVd51lh486R4tNuFaZuHdq9XaJxwnbn6eGkHvRQHBIU5ANKG9FppthLM+g+UIRTKiZrtYEMRkFfO0bxfYjX2kAYZrD9CARSjSP1Ha9OOMRDZFI44bJm8paoUIz0oQBjcj1BJ2Q5vSrQb91D387IlnjvxPe6Mpgan2yT6a40/6ha/uE+eonY8Zlt0i7EAFvyNuzC/2GqOnSgmFHGCtd9wzGe5KZv6vDC+L2c3Jx73VV0pmEsLnMGw/080aeeo3eKn8+ucsNmbI7xXE4I2j7yXARAtTzAv04SKhGnLXi5dvtzNfBZ/2qDrF65OYCP9cEC47BVkjnOy3lswozIiXPhUsIr+d2GSApB+6qo3ytxs1RTS2m/CzCuwRLz+43zYlNBPMUqQKUB2Q5v+EEO6gIm9Tx1coYnjcal34w1G9OOL7NuZfFAYESahNfsWZx4uR2OigQmMcfMyGiCwfDC9IWGNTTSlHmmsi1vKP8W9w043G9fmPgr9CDqm7tgE3UhjFfirQ8MIO3xN+SFwFhWcZeTKrf6/j13qlTlg/9K+jgcX6/p5NfIhZz4KNsE5qcPuU8f8KXpYzOiym0xLuvUmOgCagKgQNWZ9bPzVa9IDdYvTY1 LXh2cDbQ vmCoVkvcHkWvnUvSIUlYSXr+WoK9Ux+4KaFxQqmD0yaiwnsv7jVdHk5TvP82/QW0dHFbBbOF5pk2YErGCRB+AnbaZaHsis4DqlYS38j1FOhoR0V4vAa/Kepqczl8jwwhTLvSI6+wu5ADY3VjlgKGJqdiRLc52cok0DWXKv9SHbggkjghjXpY8tFyioz9U7D5kLwGh1i5PjMqbfrfJ0GHDhZ1CAteRX8fuOftTVMeMxOqkXJ7BG+v9jCD+cCAG4josDt7sIfz2e22zIhW8KDAwAkDR2w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With the ptep API sufficiently refactored, we can now introduce a new "contpte" API layer, which transparently manages the PTE_CONT bit for user mappings. Whenever it detects a set of PTEs that meet the requirements for a contiguous range, the PTEs are re-painted with the PTE_CONT bit. Use of contpte mappings is intended to be transparent to the core-mm, which continues to interact with individual ptes. Since a contpte block only has a single access and dirty bit, the semantic here changes slightly; when getting a pte (e.g. ptep_get()) that is part of a contpte mapping, the access and dirty information are pulled from the block (so all ptes in the block return the same access/dirty info). When changing the access/dirty info on a pte (e.g. ptep_set_access_flags()) that is part of a contpte mapping, this change will affect the whole contpte block. This is works fine in practice since we guarrantee that only a single folio is mapped by a contpte block, and the core-mm tracks access/dirty information per folio. This initial change provides a baseline that can be optimized in future commits. That said, fold/unfold operations (which imply tlb invalidation) are avoided where possible with a few tricks for access/dirty bit management. Write-protect modifications for contpte mappings are currently non-optimal, and incure a regression in fork() performance. This will be addressed in follow-up changes. In order for the public functions, which used to be pure inline, to continue to be callable by modules, export all the contpte_* symbols that are now called by those public inline functions. The feature is enabled/disabled with the ARM64_CONTPTE Kconfig parameter at build time. It defaults to enabled as long as its dependency, TRANSPARENT_HUGEPAGE is also enabled. The core-mm depends upon TRANSPARENT_HUGEPAGE to be able to allocate large folios, so if its not enabled, then there is no chance of meeting the physical contiguity requirement for contpte mappings. Signed-off-by: Ryan Roberts --- arch/arm64/Kconfig | 10 +- arch/arm64/include/asm/pgtable.h | 202 ++++++++++++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/contpte.c | 352 +++++++++++++++++++++++++++++++ 4 files changed, 564 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/mm/contpte.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7b071a00425d..de76e484ff3a 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2209,6 +2209,15 @@ config UNWIND_PATCH_PAC_INTO_SCS select UNWIND_TABLES select DYNAMIC_SCS +config ARM64_CONTPTE + bool "Contiguous PTE mappings for user memory" if EXPERT + depends on TRANSPARENT_HUGEPAGE + default y + help + When enabled, user mappings are configured using the PTE contiguous + bit, for any mappings that meet the size and alignment requirements. + This reduces TLB pressure and improves performance. + endmenu # "Kernel Features" menu "Boot options" @@ -2318,4 +2327,3 @@ endmenu # "CPU Power Management" source "drivers/acpi/Kconfig" source "arch/arm64/kvm/Kconfig" - diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6930c14f062f..15bc9cf1eef4 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -133,6 +133,10 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) */ #define pte_valid_not_user(pte) \ ((pte_val(pte) & (PTE_VALID | PTE_USER | PTE_UXN)) == (PTE_VALID | PTE_UXN)) +/* + * Returns true if the pte is valid and has the contiguous bit set. + */ +#define pte_valid_cont(pte) (pte_valid(pte) && pte_cont(pte)) /* * Could the pte be present in the TLB? We must check mm_tlb_flush_pending * so that we don't erroneously return false for pages that have been @@ -1116,6 +1120,202 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); +#ifdef CONFIG_ARM64_CONTPTE + +/* + * The contpte APIs are used to transparently manage the contiguous bit in ptes + * where it is possible and makes sense to do so. The PTE_CONT bit is considered + * a private implementation detail of the public ptep API (see below). + */ +extern void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); +extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); +extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr); +extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); +extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); +extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); + +static inline pte_t *contpte_align_down(pte_t *ptep) +{ + return (pte_t *)(ALIGN_DOWN((unsigned long)ptep >> 3, CONT_PTES) << 3); +} + +static inline bool contpte_is_enabled(struct mm_struct *mm) +{ + /* + * Don't attempt to apply the contig bit to kernel mappings, because + * dynamically adding/removing the contig bit can cause page faults. + * These racing faults are ok for user space, since they get serialized + * on the PTL. But kernel mappings can't tolerate faults. + */ + + return mm != &init_mm; +} + +static inline void contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * Only bother trying if both the virtual and physical addresses are + * aligned and correspond to the last entry in a contig range. The core + * code mostly modifies ranges from low to high, so this is the likely + * the last modification in the contig range, so a good time to fold. + * We can't fold special mappings, because there is no associated folio. + */ + + bool valign = ((unsigned long)ptep >> 3) % CONT_PTES == CONT_PTES - 1; + bool palign = pte_pfn(pte) % CONT_PTES == CONT_PTES - 1; + + if (contpte_is_enabled(mm) && valign && palign && + pte_valid(pte) && !pte_cont(pte) && !pte_special(pte)) + __contpte_try_fold(mm, addr, ptep, pte); +} + +static inline void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + if (contpte_is_enabled(mm) && pte_valid_cont(pte)) + __contpte_try_unfold(mm, addr, ptep, pte); +} + +/* + * The below functions constitute the public API that arm64 presents to the + * core-mm to manipulate PTE entries within the their page tables (or at least + * this is the subset of the API that arm64 needs to implement). These public + * versions will automatically and transparently apply the contiguous bit where + * it makes sense to do so. Therefore any users that are contig-aware (e.g. + * hugetlb, kernel mapper) should NOT use these APIs, but instead use the + * private versions, which are prefixed with double underscore. All of these + * APIs except for ptep_get_lockless() are expected to be called with the PTL + * held. + */ + +#define ptep_get ptep_get +static inline pte_t ptep_get(pte_t *ptep) +{ + pte_t pte = __ptep_get(ptep); + + if (!pte_valid_cont(pte)) + return pte; + + return contpte_ptep_get(ptep, pte); +} + +#define ptep_get_lockless ptep_get_lockless +static inline pte_t ptep_get_lockless(pte_t *ptep) +{ + pte_t pte = __ptep_get(ptep); + + if (!pte_valid_cont(pte)) + return pte; + + return contpte_ptep_get_lockless(ptep); +} + +static inline void set_pte(pte_t *ptep, pte_t pte) +{ + /* + * We don't have the mm or vaddr so cannot unfold or fold contig entries + * (since it requires tlb maintenance). set_pte() is not used in core + * code, so this should never even be called. Regardless do our best to + * service any call and emit a warning if there is any attempt to set a + * pte on top of an existing contig range. + */ + pte_t orig_pte = __ptep_get(ptep); + + WARN_ON_ONCE(pte_valid_cont(orig_pte)); + __set_pte(ptep, pte_mknoncont(pte)); +} + +#define set_ptes set_ptes +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + pte = pte_mknoncont(pte); + + if (!contpte_is_enabled(mm)) + __set_ptes(mm, addr, ptep, pte, nr); + else if (nr == 1) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __set_ptes(mm, addr, ptep, pte, nr); + contpte_try_fold(mm, addr, ptep, pte); + } else + contpte_set_ptes(mm, addr, ptep, pte, nr); +} + +static inline void pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __pte_clear(mm, addr, ptep); +} + +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +static inline pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + return __ptep_get_and_clear(mm, addr, ptep); +} + +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG +static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_valid_cont(orig_pte)) + return __ptep_test_and_clear_young(vma, addr, ptep); + + return contpte_ptep_test_and_clear_young(vma, addr, ptep); +} + +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +static inline int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_valid_cont(orig_pte)) + return __ptep_clear_flush_young(vma, addr, ptep); + + return contpte_ptep_clear_flush_young(vma, addr, ptep); +} + +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +static inline void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __ptep_set_wrprotect(mm, addr, ptep); + contpte_try_fold(mm, addr, ptep, __ptep_get(ptep)); +} + +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +static inline int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + pte_t orig_pte = __ptep_get(ptep); + + entry = pte_mknoncont(entry); + + if (!pte_valid_cont(orig_pte)) + return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + + return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); +} + +#else /* CONFIG_ARM64_CONTPTE */ + #define ptep_get __ptep_get #define set_pte __set_pte #define set_ptes __set_ptes @@ -1131,6 +1331,8 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#endif /* CONFIG_ARM64_CONTPTE */ + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d..60454256945b 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,6 +3,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o fixmap.o +obj-$(CONFIG_ARM64_CONTPTE) += contpte.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c new file mode 100644 index 000000000000..e079ec61d7d1 --- /dev/null +++ b/arch/arm64/mm/contpte.c @@ -0,0 +1,352 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2023 ARM Ltd. + */ + +#include +#include +#include + +static void ptep_clear_flush_range(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, int nr) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long start_addr = addr; + int i; + + for (i = 0; i < nr; i++, ptep++, addr += PAGE_SIZE) + __pte_clear(mm, addr, ptep); + + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); +} + +static bool ptep_any_valid(pte_t *ptep, int nr) +{ + int i; + + for (i = 0; i < nr; i++, ptep++) { + if (pte_valid(__ptep_get(ptep))) + return true; + } + + return false; +} + +static void contpte_convert(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long start_addr; + pte_t *start_ptep; + int i; + + start_ptep = ptep = contpte_align_down(ptep); + start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), CONT_PTES), pte_pgprot(pte)); + + for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) { + pte_t ptent = __ptep_get_and_clear(mm, addr, ptep); + + if (pte_dirty(ptent)) + pte = pte_mkdirty(pte); + + if (pte_young(ptent)) + pte = pte_mkyoung(pte); + } + + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); + + __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); +} + +void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * We have already checked that the virtual and pysical addresses are + * correctly aligned for a contpte mapping in contpte_try_fold() so the + * remaining checks are to ensure that the contpte range is fully + * covered by a single folio, and ensure that all the ptes are valid + * with contiguous PFNs and matching prots. We ignore the state of the + * access and dirty bits for the purpose of deciding if its a contiguous + * range; the folding process will generate a single contpte entry which + * has a single access and dirty bit. Those 2 bits are the logical OR of + * their respective bits in the constituent pte entries. In order to + * ensure the contpte range is covered by a single folio, we must + * recover the folio from the pfn, but special mappings don't have a + * folio backing them. Fortunately contpte_try_fold() already checked + * that the pte is not special - we never try to fold special mappings. + * Note we can't use vm_normal_page() for this since we don't have the + * vma. + */ + + struct page *page = pte_page(pte); + struct folio *folio = page_folio(page); + unsigned long folio_saddr = addr - (page - &folio->page) * PAGE_SIZE; + unsigned long folio_eaddr = folio_saddr + folio_nr_pages(folio) * PAGE_SIZE; + unsigned long cont_saddr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + unsigned long cont_eaddr = cont_saddr + CONT_PTE_SIZE; + unsigned long pfn; + pgprot_t prot; + pte_t subpte; + pte_t *orig_ptep; + int i; + + if (folio_saddr > cont_saddr || folio_eaddr < cont_eaddr) + return; + + pfn = pte_pfn(pte) - ((addr - cont_saddr) >> PAGE_SHIFT); + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + orig_ptep = ptep; + ptep = contpte_align_down(ptep); + + for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) { + subpte = __ptep_get(ptep); + subpte = pte_mkold(pte_mkclean(subpte)); + + if (!pte_valid(subpte) || + pte_pfn(subpte) != pfn || + pgprot_val(pte_pgprot(subpte)) != pgprot_val(prot)) + return; + } + + pte = pte_mkcont(pte); + contpte_convert(mm, addr, orig_ptep, pte); +} +EXPORT_SYMBOL(__contpte_try_fold); + +void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * We have already checked that the ptes are contiguous in + * contpte_try_unfold(), so we can unfold unconditionally here. + */ + + pte = pte_mknoncont(pte); + contpte_convert(mm, addr, ptep, pte); +} +EXPORT_SYMBOL(__contpte_try_unfold); + +pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte) +{ + /* + * Gather access/dirty bits, which may be populated in any of the ptes + * of the contig range. We are guarranteed to be holding the PTL, so any + * contiguous range cannot be unfolded or otherwise modified under our + * feet. + */ + + pte_t pte; + int i; + + ptep = contpte_align_down(ptep); + + for (i = 0; i < CONT_PTES; i++, ptep++) { + pte = __ptep_get(ptep); + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} +EXPORT_SYMBOL(contpte_ptep_get); + +pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) +{ + /* + * Gather access/dirty bits, which may be populated in any of the ptes + * of the contig range. We may not be holding the PTL, so any contiguous + * range may be unfolded/modified/refolded under our feet. Therefore we + * ensure we read a _consistent_ contpte range by checking that all ptes + * in the range are valid and have CONT_PTE set, that all pfns are + * contiguous and that all pgprots are the same (ignoring access/dirty). + * If we find a pte that is not consistent, then we must be racing with + * an update so start again. If the target pte does not have CONT_PTE + * set then that is considered consistent on its own because it is not + * part of a contpte range. + */ + + pte_t orig_pte; + pgprot_t orig_prot; + pte_t *ptep; + unsigned long pfn; + pte_t pte; + pgprot_t prot; + int i; + +retry: + orig_pte = __ptep_get(orig_ptep); + + if (!pte_valid_cont(orig_pte)) + return orig_pte; + + orig_prot = pte_pgprot(pte_mkold(pte_mkclean(orig_pte))); + ptep = contpte_align_down(orig_ptep); + pfn = pte_pfn(orig_pte) - (orig_ptep - ptep); + + for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) { + pte = __ptep_get(ptep); + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + + if (!pte_valid_cont(pte) || + pte_pfn(pte) != pfn || + pgprot_val(prot) != pgprot_val(orig_prot)) + goto retry; + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} +EXPORT_SYMBOL(contpte_ptep_get_lockless); + +void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + unsigned long next; + unsigned long end = addr + (nr << PAGE_SHIFT); + unsigned long pfn = pte_pfn(pte); + pgprot_t prot = pte_pgprot(pte); + pte_t orig_pte; + + do { + next = pte_cont_addr_end(addr, end); + nr = (next - addr) >> PAGE_SHIFT; + pte = pfn_pte(pfn, prot); + + if (((addr | next | (pfn << PAGE_SHIFT)) & ~CONT_PTE_MASK) == 0) + pte = pte_mkcont(pte); + else + pte = pte_mknoncont(pte); + + /* + * If operating on a partial contiguous range then we must first + * unfold the contiguous range if it was previously folded. + * Otherwise we could end up with overlapping tlb entries. + */ + if (nr != CONT_PTES) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + /* + * If we are replacing ptes that were contiguous or if the new + * ptes are contiguous and any of the ptes being replaced are + * valid, we need to clear and flush the range to prevent + * overlapping tlb entries. + */ + orig_pte = __ptep_get(ptep); + if (pte_valid_cont(orig_pte) || + (pte_cont(pte) && ptep_any_valid(ptep, nr))) + ptep_clear_flush_range(mm, addr, ptep, nr); + + __set_ptes(mm, addr, ptep, pte, nr); + + addr = next; + ptep += nr; + pfn += nr; + + } while (addr != end); +} +EXPORT_SYMBOL(contpte_set_ptes); + +int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + /* + * ptep_clear_flush_young() technically requires us to clear the access + * flag for a _single_ pte. However, the core-mm code actually tracks + * access/dirty per folio, not per page. And since we only create a + * contig range when the range is covered by a single folio, we can get + * away with clearing young for the whole contig range here, so we avoid + * having to unfold. + */ + + int i; + int young = 0; + + ptep = contpte_align_down(ptep); + addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + + for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) + young |= __ptep_test_and_clear_young(vma, addr, ptep); + + return young; +} +EXPORT_SYMBOL(contpte_ptep_test_and_clear_young); + +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + int young; + + young = contpte_ptep_test_and_clear_young(vma, addr, ptep); + + if (young) { + /* + * See comment in __ptep_clear_flush_young(); same rationale for + * eliding the trailing DSB applies here. + */ + addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + __flush_tlb_range_nosync(vma, addr, addr + CONT_PTE_SIZE, + PAGE_SIZE, true, 3); + } + + return young; +} +EXPORT_SYMBOL(contpte_ptep_clear_flush_young); + +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + pte_t orig_pte; + int i; + unsigned long start_addr; + + /* + * Gather the access/dirty bits for the contiguous range. If nothing has + * changed, its a noop. + */ + orig_pte = ptep_get(ptep); + if (pte_val(orig_pte) == pte_val(entry)) + return 0; + + /* + * We can fix up access/dirty bits without having to unfold/fold the + * contig range. But if the write bit is changing, we need to go through + * the full unfold/fold cycle. + */ + if (pte_write(orig_pte) == pte_write(entry)) { + /* + * For HW access management, we technically only need to update + * the flag on a single pte in the range. But for SW access + * management, we need to update all the ptes to prevent extra + * faults. Avoid per-page tlb flush in __ptep_set_access_flags() + * and instead flush the whole range at the end. + */ + ptep = contpte_align_down(ptep); + start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + + for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) + __ptep_set_access_flags(vma, addr, ptep, entry, 0); + + if (dirty) + __flush_tlb_range(vma, start_addr, addr, + PAGE_SIZE, true, 3); + } else { + __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); + __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + contpte_try_fold(vma->vm_mm, addr, ptep, entry); + } + + return 1; +} +EXPORT_SYMBOL(contpte_ptep_set_access_flags); From patchwork Mon Dec 4 10:54:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4DCDC4167B for ; Mon, 4 Dec 2023 10:55:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 310E06B02B6; Mon, 4 Dec 2023 05:55:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C0F56B02B8; Mon, 4 Dec 2023 05:55:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13B436B02B9; Mon, 4 Dec 2023 05:55:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 02D7D6B02B6 for ; Mon, 4 Dec 2023 05:55:50 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D718D1401AE for ; Mon, 4 Dec 2023 10:55:49 +0000 (UTC) X-FDA: 81528830418.27.2A99CB0 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf23.hostedemail.com (Postfix) with ESMTP id 380A3140003 for ; Mon, 4 Dec 2023 10:55:48 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687348; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w1WNdKqA4Y1Q1WabMoagwRQ3JEZGGW7XgKYZUGZQZTE=; b=PDH3t+t0hBKZiiM4awfBCpI07SRRmkuFJNVXpQ4OkcyWPNmIpziCGYMarXDdsSqC/Q8U7r eQbJ1Vq3tGRg9YPBLTd5dsec0rdWrgWnMFqpAZZ5XO9GE94ssMWW8eDx+iTGiJ6qyOE954 2bHKprfWvNckVBs97sAl/PPz10wEnZE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf23.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687348; a=rsa-sha256; cv=none; b=cMfWDFcJQQAaOCcyVV15XzVO/+dFvEWx7WsZ5Hf4idDVzsKIDi1h3PRa0D8v14HY8d219R IsJXIatpyBTTB/N+uDS4qyeTuTIsbR9U2XX2KpOc4/8wCpechbbL19kFA3u2AcXqFsF5b3 N1P1mAUcvGX0m6rZx6Vbl2j04x17N+8= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D6749139F; Mon, 4 Dec 2023 02:56:34 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2EF503F6C4; Mon, 4 Dec 2023 02:55:44 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 14/15] arm64/mm: Implement ptep_set_wrprotects() to optimize fork() Date: Mon, 4 Dec 2023 10:54:39 +0000 Message-Id: <20231204105440.61448-15-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 380A3140003 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1xdkyrey6qn4f9ida51nm4db4fdhendi X-HE-Tag: 1701687348-679776 X-HE-Meta: U2FsdGVkX1+AU4jYHp2SmgwkLEGeXP6cM1hQJl7MyAKnvkFavNfJtnjcNCqxv9lc+CdhCUF+ACeQFUyvHpfmFJ496qPZKW7fDd0JHL/dTlMQtnLeK1cTHIDP8aIu6QFGp9OmC/vdvD3/x/VoSb+CjXyJvbqU9vO8z+iHmpbC95/2Vo6wXLW9WjaP1NWgeZGU5g5IQJLnbasFWmLkUTIPkNWKXYQlg1hLtXe/qlp4X/Ai7DltA0ReRY9m8go2yDVBCmDl7CUM52+Wmv+1EgqLn6CCgmocPlbKWw50gBAQ/MvKE53pKY5fJKjwBNRBvNandGKQT9bUIISF42+7FTNbP2JlMu7XKNmK7j66rfLGguDO4LFvMsIw4PU5YcSmHsJpfwbQzc03QMUEpGNiI1xPTZHswHhuHSv6w/CjU3ZbB8kvHicz1kJVEr+J5QhhHdsPXTwBdHRMxUBYoHw/GcUhOEuwC7XD4sPc41UUHkQ5mme6Nu/9IBexYCMZd4b6ixGhIlk+PWFIv/wUUM5RmxGOG3XqKjqD5GA35yslpADVFYw5QqtH1XXE1dByq8nnUFL20eFLZX3cGVA7z/BJM1qQxRRqZgohZDY7hA6oiRcyNDrTTJ3vK44oZXwHJUkpXJy6L/I/ZLwFiiEAkyo70TctT6woBPTFsxF2YeSXPauomRBXlVBYlMe8fDitrS7o1hdYwQwBpxJ3hs2JrNQbb4fTXNCmNnSKLYHiKQrvLCw0cYx+SJQVw9DEogryZesxvOghMLNwA0k62ehSluZtxqD0Fv3LoGygd03YWpZWkpbHZSoa38QXzjeOft/TUOs0B+6vSAGiN0izRqSymFnujQO8sQJ45eSed7I4X0rv5OFvMfLBAVdCSQJ1/0cyObVVnffAViiRFoJZMLLzjsCyQQuMirQTk1eqJeOt3AsWSIilWoXQdFztv6TxZC1BH3t/tuR6CeTFmFy7XsqZ5P+jkRw BMm0OEMv 83kUeIs5yJ8Pt5yr1s/BepzCLeN1BapheMH5ewkPIslwmIJWDuEhGAlenO/7ZNIx99IeOorol2qAOP8/TidF49VzAhaItcXk5YbLfGmYsjzBoMCEKuembxG18wxW+hG98/pIqUHs1S5uotRBHqJ+rZsA1fFQJ/6SBNMSgEbs0zA57OeCmWutSEaqq7waa4fS0ruZwUqNeu8GFNfWPm9HI8gJMw3Joqnrbp5807+DhLNiOugFcBIBtsqwNswiykXcrzbi+qEbwUlLaCjZ7dga5rWH1kg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With the core-mm changes in place to batch-copy ptes during fork, we can take advantage of this in arm64 to greatly reduce the number of tlbis we have to issue, and recover the lost fork performance incured when adding support for transparent contiguous ptes. If we are write-protecting a whole contig range, we can apply the write-protection to the whole range and know that it won't change whether the range should have the contiguous bit set or not. For ranges smaller than the contig range, we will still have to unfold, apply the write-protection, then fold if the change now means the range is foldable. This optimization is possible thanks to the tightening of the Arm ARM in respect to the definition and behaviour when 'Misprogramming the Contiguous bit'. See section D21194 at https://developer.arm.com/documentation/102105/latest/ Performance tested with the following test written for the will-it-scale framework: ------- char *testcase_description = "fork and exit"; void testcase(unsigned long long *iterations, unsigned long nr) { int pid; char *mem; mem = malloc(SZ_128M); assert(mem); memset(mem, 1, SZ_128M); while (1) { pid = fork(); assert(pid >= 0); if (!pid) exit(0); waitpid(pid, NULL, 0); (*iterations)++; } } ------- I see huge performance regression when PTE_CONT support was added, then the regression is mostly fixed with the addition of this change. The following shows regression relative to before PTE_CONT was enabled (bigger negative value is bigger regression): | cpus | before opt | after opt | |-------:|-------------:|------------:| | 1 | -10.4% | -5.2% | | 8 | -15.4% | -3.5% | | 16 | -38.7% | -3.7% | | 24 | -57.0% | -4.4% | | 32 | -65.8% | -5.4% | Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 30 ++++++++++++++++++++--- arch/arm64/mm/contpte.c | 42 ++++++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 15bc9cf1eef4..9bd2f57a9e11 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -984,6 +984,16 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, } while (pte_val(pte) != pte_val(old_pte)); } +static inline void __ptep_set_wrprotects(struct mm_struct *mm, + unsigned long address, pte_t *ptep, + unsigned int nr) +{ + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + __ptep_set_wrprotect(mm, address, ptep); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, @@ -1139,6 +1149,8 @@ extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +extern void contpte_set_wrprotects(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); @@ -1290,13 +1302,25 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return contpte_ptep_clear_flush_young(vma, addr, ptep); } +#define ptep_set_wrprotects ptep_set_wrprotects +static inline void ptep_set_wrprotects(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + if (!contpte_is_enabled(mm)) + __ptep_set_wrprotects(mm, addr, ptep, nr); + else if (nr == 1) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __ptep_set_wrprotects(mm, addr, ptep, 1); + contpte_try_fold(mm, addr, ptep, __ptep_get(ptep)); + } else + contpte_set_wrprotects(mm, addr, ptep, nr); +} + #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __ptep_set_wrprotect(mm, addr, ptep); - contpte_try_fold(mm, addr, ptep, __ptep_get(ptep)); + ptep_set_wrprotects(mm, addr, ptep, 1); } #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index e079ec61d7d1..2a57df16bf58 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -303,6 +303,48 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, } EXPORT_SYMBOL(contpte_ptep_clear_flush_young); +void contpte_set_wrprotects(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + unsigned long next; + unsigned long end = addr + (nr << PAGE_SHIFT); + + do { + next = pte_cont_addr_end(addr, end); + nr = (next - addr) >> PAGE_SHIFT; + + /* + * If wrprotecting an entire contig range, we can avoid + * unfolding. Just set wrprotect and wait for the later + * mmu_gather flush to invalidate the tlb. Until the flush, the + * page may or may not be wrprotected. After the flush, it is + * guarranteed wrprotected. If its a partial range though, we + * must unfold, because we can't have a case where CONT_PTE is + * set but wrprotect applies to a subset of the PTEs; this would + * cause it to continue to be unpredictable after the flush. + */ + if (nr != CONT_PTES) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + __ptep_set_wrprotects(mm, addr, ptep, nr); + + addr = next; + ptep += nr; + + /* + * If applying to a partial contig range, the change could have + * made the range foldable. Use the last pte in the range we + * just set for comparison, since contpte_try_fold() only + * triggers when acting on the last pte in the contig range. + */ + if (nr != CONT_PTES) + contpte_try_fold(mm, addr - PAGE_SIZE, ptep - 1, + __ptep_get(ptep - 1)); + + } while (addr != end); +} +EXPORT_SYMBOL(contpte_set_wrprotects); + int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty) From patchwork Mon Dec 4 10:54:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66A8CC4167B for ; Mon, 4 Dec 2023 10:55:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F205C6B02B8; Mon, 4 Dec 2023 05:55:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ECF566B02BA; Mon, 4 Dec 2023 05:55:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D970B6B02BB; Mon, 4 Dec 2023 05:55:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CAC256B02B8 for ; Mon, 4 Dec 2023 05:55:53 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9BA0E1601A5 for ; Mon, 4 Dec 2023 10:55:53 +0000 (UTC) X-FDA: 81528830586.30.5F19EFE Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id DC66D4000A for ; Mon, 4 Dec 2023 10:55:51 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687352; a=rsa-sha256; cv=none; b=csZ6zXc2022UUhmeFY7GukymaHqH9bCJr7tvXMYHQAx6KxaSRpDgkKeMwPC6GpEMH8G1Ty Wk/Ss1kqsWvMvPrmh357PEH6OeNwnp62x5/6ZDwVfVnTMx4Dehg+yxAZdIG+yBd6e0vpM6 aMbgNetKtP2pzhM1U2fL1XYmLrxIW6k= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687352; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+nCRyAXXqihs+BnNmeqcBD6lxH4FGaiu3Zd52hGhiqQ=; b=8bCMyQm6F3pB8H4HVUM1OdLdTSOKInIoF87q+FVfzpchXRSeT1khE9TkhDwABKIavrMODG G1LaDP/la3tRYly3L7JOJeNjK3g87v++OkjEJFgcfikF3fsMGsM6iD85HQ9RvSpwrGMsri qZn1s0PN+gCdH3LI+LbeLFZWz8YCxAY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7ECBC1650; Mon, 4 Dec 2023 02:56:38 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C9E2E3F6C4; Mon, 4 Dec 2023 02:55:47 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 15/15] arm64/mm: Implement clear_ptes() to optimize exit() Date: Mon, 4 Dec 2023 10:54:40 +0000 Message-Id: <20231204105440.61448-16-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DC66D4000A X-Stat-Signature: 8jkt9aw8rasmpw3ajjdb8s3bf9extxbp X-Rspam-User: X-HE-Tag: 1701687351-845665 X-HE-Meta: U2FsdGVkX18WGzbY5UWUi1Qh7GJDhOqrHbAw/hoGj/cMitUfxPcqUBO0tspWfwIQ4lecenzCskDEqFipgucolYwOrSlhDbbxoRMwM9vkiAV9QP3rOPpEWBXl92PO5GDqVMLI/cW3CmvODaESFWnlq5EJMKCr5AHjrsz6EWzlwlhi470163We0ioqJzjwlNJ93qJK9jPOku88hCx5JbxGVC9RbXS3woK6+lubXGyeBPVkN7TsxyYMqojPxzoMjaDmQSCTFBp4ClCnFSEgMHLSk3yZhxPK9d9N3RK4VPQ5RnUk3UAL077UEVZZ+Hl4nx3sgDn0ao2ZdwlUBJgIPUWa/6Yp3rxUbOLL7CHxMcP/RXrG7GcAXX49U4aSrZUjCaY78EaEvo0/jDqDhvW15Gtem4UW/FWSBddU3LZc8ZqhAyR7WKeFqpXQC91VRwiXS/62Ug0WTF7QvfUY1rUJPj1DmxYVkVNPwQTXR/EWJNOnU8yaYOYVj2lT1KqraKT8Y9MOS/tBl7r0qxBIK8f7o/1EkTrs+H3TQYGZGBH9AZeyv0PxCgAwvgl3/S2Iy3RHD0pIIF6OaH3GLnV6mip4kaKZg1gdb3S+Db7E/noUCAzCeymTvE+DHxoBPV/sNSMyVgweb3gt8Ew3n3ZsK+jHsjvr7ZfCD9gisvfRx6V+EVnwU9Bt6Hd1jN07vXPJ35Q9Y0PwsZv8BLLSVqczwFzGxnEntHZhWXW36CaejSqFUJO73uPX1RXGi17WfVRibl0bLMUTLij3iXq2iQAye15vdgGoSBNG0/KRo5rx8YDQSRKnIDX14mntehOE23gdb9o4JicbdeYM0balgK2G3+yJHlmI6jW2hCt6o5wR5quM360vP+gLKzoi38U+RIdeeD/CXvAMeOowrBog70PlEx+lpBXmlYan5saxu1gtOGvDFRUhpfSeTU7YVcaMSjIwUIx74IYcOulpMQuuNJY5I44/jM6 nNOC5j+h NgteGk4+lO6sABaYC2WsrYoLdEiPsF6hNEs4u4lnMPCCDAbi/v3Bb+7F4+CqZzlyk9vzURxIIcF1plt4CrCAGK4PG7McB2CricTL1HirotQo2d2o0rMglgMkd1OQvS6aEH3URu9tjiVKevsQ2mToD1UcnQ7WhBJxXWt+e2lv2PAqwvZ1kGUTRAzl3PyhnISPrP9ejHXYOI/o9ezlQ3ncNAmgVUQybFoONJ3xpgf/Q/x8ABtX8ubYuTk2IL585VWC500LirYyQtMUI065igkjLb0QvTg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With the core-mm changes in place to batch-clear ptes during zap_pte_range(), we can take advantage of this in arm64 to greatly reduce the number of tlbis we have to issue, and recover the lost exit performance incured when adding support for transparent contiguous ptes. If we are clearing a whole contpte range, we can elide first unfolding that range and save the tlbis. We just clear the whole range. The following shows the results of running a kernel compilation workload and measuring the cost of arm64_sys_exit_group() (which at ~1.5% is a very small part of the overall workload). Benchmarks were run on Ampere Altra in 2 configs; single numa node and 2 numa nodes (tlbis are more expensive in 2 node config). - baseline: v6.7-rc1 + anonfolio-v7 - no-opt: contpte series without any attempt to optimize exit() - simple-ptep_get_clear_full: simple optimization to exploit full=1. ptep_get_clear_full() does not fully conform to its intended semantic - robust-ptep_get_clear_full: similar to previous but ptep_get_clear_full() fully conforms to its intended semantic - clear_ptes: optimization implemented by this patch | config | numa=1 | numa=2 | |----------------------------|--------|--------| | baseline | 0% | 0% | | no-opt | 190% | 768% | | simple-ptep_get_clear_full | 8% | 29% | | robust-ptep_get_clear_full | 21% | 19% | | clear_ptes | 13% | 9% | In all cases, the cost of arm64_sys_exit_group() increases; this is anticipated because there is more work to do to tear down the page tables. But clear_ptes() gives the smallest increase overall. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 32 ++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 42 ++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9bd2f57a9e11..ff6b3cc9e819 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1145,6 +1145,8 @@ extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); +extern pte_t contpte_clear_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, @@ -1270,6 +1272,36 @@ static inline void pte_clear(struct mm_struct *mm, __pte_clear(mm, addr, ptep); } +#define clear_ptes clear_ptes +static inline pte_t clear_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, int full, + unsigned int nr) +{ + pte_t pte; + + if (!contpte_is_enabled(mm)) { + unsigned int i; + pte_t tail; + + pte = __ptep_get_and_clear(mm, addr, ptep); + for (i = 1; i < nr; i++) { + addr += PAGE_SIZE; + ptep++; + tail = __ptep_get_and_clear(mm, addr, ptep); + if (pte_dirty(tail)) + pte = pte_mkdirty(pte); + if (pte_young(tail)) + pte = pte_mkyoung(pte); + } + } else if (nr == 1) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + pte = __ptep_get_and_clear(mm, addr, ptep); + } else + pte = contpte_clear_ptes(mm, addr, ptep, nr); + + return pte; +} + #define __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 2a57df16bf58..34b43bde3fcd 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -257,6 +257,48 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL(contpte_set_ptes); +pte_t contpte_clear_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + unsigned int nr) +{ + /* + * If we cover a partial contpte block at the beginning or end of the + * batch, unfold if currently folded. This makes it safe to clear some + * of the entries while keeping others. contpte blocks in the middle of + * the range, which are fully covered don't need to be unfolded because + * we will clear the full block. + */ + + unsigned int i; + pte_t pte; + pte_t tail; + + if (ptep != contpte_align_down(ptep) || nr < CONT_PTES) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + if (ptep + nr != contpte_align_down(ptep + nr)) + contpte_try_unfold(mm, addr + PAGE_SIZE * (nr - 1), + ptep + nr - 1, + __ptep_get(ptep + nr - 1)); + + pte = __ptep_get_and_clear(mm, addr, ptep); + + for (i = 1; i < nr; i++) { + addr += PAGE_SIZE; + ptep++; + + tail = __ptep_get_and_clear(mm, addr, ptep); + + if (pte_dirty(tail)) + pte = pte_mkdirty(pte); + + if (pte_young(tail)) + pte = pte_mkyoung(pte); + } + + return pte; +} +EXPORT_SYMBOL(contpte_clear_ptes); + int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) {