From patchwork Wed May 8 19:19:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 971BFC04FFE for ; Wed, 8 May 2024 19:30:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2EA096B00A9; Wed, 8 May 2024 15:30:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2722A6B00AA; Wed, 8 May 2024 15:30:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EC6D6B00AB; Wed, 8 May 2024 15:30:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DE9956B00A9 for ; Wed, 8 May 2024 15:30:52 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A04E941117 for ; Wed, 8 May 2024 19:30:52 +0000 (UTC) X-FDA: 82096221144.24.7B03E29 Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) by imf14.hostedemail.com (Postfix) with ESMTP id 71ACB10001B for ; Wed, 8 May 2024 19:30:50 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=hJogDrsk; dmarc=none; spf=pass (imf14.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.208.179 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196650; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; b=SBbYYvkesewkw05bJXGNANkPaWIRy+AVHHrN8PBjPp3cSjve/qO2yr6t6tmFKfSSVicHKF t3hb6nHsnsi0cTR41uoa9Spmx98/M0h+Cefsk6f6xXMTSTs/1SX+V5nMt9StDwPuScJcYM w81S3y2s6C7XLOcF6EP2zzuwRXlZEQQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=hJogDrsk; dmarc=none; spf=pass (imf14.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.208.179 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196650; a=rsa-sha256; cv=none; b=vvPMMLHgjClrwusAGssQapsT6MXkRmsdF+wbMH+yV/+n55y/RWquUtylbAvzNii3dX3B7z RDDndcWxU/BAHHzNcL8VnM6Mqj775Tod4MznNHYkjYHQJacNFYJQ0IiB/AVmA96ugz0pFl 0iKod+1XBTSnzHeXOkmIuua9lQ66Eco= Received: by mail-lj1-f179.google.com with SMTP id 38308e7fff4ca-2e0a34b2899so1964781fa.3 for ; Wed, 08 May 2024 12:30:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196649; x=1715801449; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; b=hJogDrskHnX1/5z7ekXgMlOhQVLi7/DwlCBuvyTWoe9F/jxN4tAUjI6V1wN3OhJ0SZ v9qUNRDce4W/BlKAKVo8uQrQvNv5bXxOQtTMcxI6rXMZIKbj5zUSReNkHTcNPxtC1wqK a2VqoNUOeRyaDEUtCy//KmX1vym4ru4TiLKgmEyyXHckQIj3nWDemkA5EyGWI7Ai97/r 1J8LVBySiIhxaMxLQmotUwudh+cDalm9g31IW+lKYy7e49VpUc+taPrs7ZlwawnbK2Jk C4OG33ASKBa+HK2YOBWW+KLRH4+WQesJ3SEzgBp1yMh3Z/BsYl9rl3VBOnANB4kqu578 jODQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196649; x=1715801449; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; b=EILrzQT/t+wANSYH+eL5N6Qnnly5p9FkUjuOfXud7hpNxiICaTTGP4SFxJyXduSnsx laYwgrzdGKW8uCiWNiRgG4c9V5c7uz1jF3b8oQbMoyUkY71FS+NlWK8vGCZ6HWlewPlD D+QDZABmmc1J5BLuwSxcw2cI6sT1MU3SZUHjHFHteaqbieU/OuHkvd/NxDwBOnhCurwn EtR3GSHtpdoBrSeRITbB9SZvfhZH/P+U0h+gQfl1bpecwhDwMK9ygiPWOwY2xtwYkPub E1RHxYlazCPGDfcQeSAeQo2aMYX5mA/BCBXzXcnN7TzcgaeVNBkb/E9db+mR532dza0k 3ukg== X-Forwarded-Encrypted: i=1; AJvYcCUw+jC9RnVSfEO12imx1md2Ds4Hchk+TZa6CiqQWorhN6FosZiEGRzbe0BGD9DwoV646g4GTsqE7q+wh3H2RnHaaBc= X-Gm-Message-State: AOJu0YxS+MorfKuC56WOxmIVxNaSk4lnaUFyVeNqNCtTgtqCxO1aZUeZ cmKC/5br+x5SA3dAoa3dR5XMvKce4Xm+NCXan5BnaLXLm/8YPLuZHgDWWUcuApg= X-Google-Smtp-Source: AGHT+IGXrMpV3e8NWTuY3a54mvQHDm5XG0kBaIk0g0HXiaoXOXDbgZBz/5WZyQXp8WNGVe1a2PspPg== X-Received: by 2002:a05:651c:1541:b0:2df:e192:47ec with SMTP id 38308e7fff4ca-2e447081ef2mr37890971fa.29.1715196648606; Wed, 08 May 2024 12:30:48 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id b15-20020a05600c4e0f00b0041aa79f27a0sm3273819wmq.38.2024.05.08.12.30.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:30:48 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 11/12] mm, riscv, arm64: Use common ptep_set_wrprotect()/wrprotect_ptes() functions Date: Wed, 8 May 2024 21:19:30 +0200 Message-Id: <20240508191931.46060-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Stat-Signature: pyftaysm11q9qbzuege1zidtqykuptdg X-Rspam-User: X-Rspamd-Queue-Id: 71ACB10001B X-HE-Tag: 1715196650-501329 X-HE-Meta: U2FsdGVkX18+0H7SsV7kXRWzDGEMe0wUvRAfg0fbiARQMt0SYTCcExJhNVimkUNHzZWNPKY5E5d5bIBulw03KcVZRiFhADdhO+Wfs7z/NlvDkCsrMIdhFnAxHx4IhCmobX6pxOBL2e+XA0oa/9/DxWdSSSY/V5phsZxrfgHj5PRsXH2xJek3sjfOqZW2TwFNjuwK5t/dOEB6XJiIYCKox9sb4IBzJ7BDNgOdc0KBdheX7N//KIp3yqI+hnSp636OlgtduZ7Y24qrdvhA+6H42gXS51cDsTYOWuPpg0s9sv2ee4u3yUC20MsEDMPDls7doxRf1ryuQv1dwplszq25XIueSpTuYnQ+p1XzLcmATyJqxoj5sKIyUxtbk6bEVo0QoZHd1ygBIdq8I4c6jWkYyl2pUEyvbu1SiFF4GjLxT9tu/NSp3rmBOh+VxIqkNrg8uvS5VUvYKagsOrC6PNhfR3LOycC08mtLkKkuOZ9Sr16bewIfOLF90JapSdIhaSyQHkqVh8YLQk9mpdONtiZuXsSlqf7tgvIObRrJvD8sxA/qpR+f2BnOw3uHYnanFu3yFCRlkmYYZMByMvqjOkypb0p/mU8O9fu/GaXDFxYdV7rVnWXY9qmB7U6V2EplI6wPqcgmzBjQ0fTPL/2jXQj0ozDYHtSabK0xwePZ6zDcCTDR8VopMPFh7xIciXDZgjPN3FSqqENn1n803d/KJadJzqdnf3AyHyvDRAbfV9HlGYYCE6SCdlkHQirIkd00Q+ZpxYWAnTLTGSeR3Nb78C4sgMagFBpvxZsNNjlbOnyKxM+wBjoG4xc6afYBxhZo/BJnKEG6VrwBACrRZ9sbOyT4pQpHTc/0lG7brDN21DvqX3WBNXE3ze3aWD+QkW0eEmJKEb8yl8wKag4kQ5BzSx7kSA0a5FdnRcUvEcfi52k6bmlrd8/prRe+AbU5ejb6EbsaliDIIiFCx6lJhapn4gh 9S7g7sHX zQny/5q174E5fCgd2HsAADk1NkSVACUxzae8n25uer+UPL4dRrfMpDphi4TzSc1K9npZ4J6hDtSEjtOpfi+Yrpt4igQs6BfE/s8Ol6ck6b1NtuTQUwQmN8zlSBMcRdt4nBXtd9m1wv/Tdid9ZW+3+911j2udbuVwU6HW19D+Rt33maSRxvT7ZIxwN0624gKU2/TkZK/KPsMLK6hclwZdfbwvCeSem49Dr6GVbfpWryhgdCXsxI3lcYrbbSrlw5m8wIDN+sE9I0ndDFhV1yi3lq4pSrigclHVxfTtcFFLqxJvsWmHHG353t5zHoBM0aCYhmcaNY/k1Q5kT3mlGD2axRArJqJvVEAZvKKVDOVHiq/88xRAnZ12eP0Rq2HqDCMi6xjYLu6DexTaMeYLw365IzE9aDq9h05jRj9v7n9RN35vCDCudV2BFxQB4AU2D+cvMggoS42Ge4G7kRA6wRPJe5ioMSzYgmUdElocUiD8wA+0xS5yv4H7p/6kwFYgxVDrWdUcCLO52E5NRxbKcnFMR1X2CQiOKOIVopBAs9k36cAmQCTg9XR1HbeJo+g+mPxILy+ou X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware ptep_set_wrprotect()/wrprotect_ptes() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 56 ++++++------------------ arch/arm64/mm/contpte.c | 18 -------- arch/riscv/include/asm/pgtable.h | 25 +++++++++-- include/linux/contpte.h | 2 + mm/contpte.c | 75 +++++++++++++++++++++++++++++++- 5 files changed, 110 insertions(+), 66 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6591aab11c67..162efd9647dd 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1208,7 +1208,11 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline void ___ptep_set_wrprotect(struct mm_struct *mm, +/* + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware + * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. + */ +static inline void __ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep, pte_t pte) { @@ -1222,23 +1226,13 @@ static inline void ___ptep_set_wrprotect(struct mm_struct *mm, } while (pte_val(pte) != pte_val(old_pte)); } -/* - * __ptep_set_wrprotect - mark read-only while trasferring potential hardware - * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. - */ -static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) -{ - ___ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); -} - static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, pte_t *ptep, unsigned int nr) { unsigned int i; for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) - __ptep_set_wrprotect(mm, address, ptep); + __ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -1246,7 +1240,7 @@ static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - __ptep_set_wrprotect(mm, address, (pte_t *)pmdp); + __ptep_set_wrprotect(mm, address, (pte_t *)pmdp, __ptep_get((pte_t *)pmdp)); } #define pmdp_establish pmdp_establish @@ -1389,8 +1383,6 @@ extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); -extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr); #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) @@ -1478,35 +1470,12 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); #define wrprotect_ptes wrprotect_ptes -static __always_inline void wrprotect_ptes(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, unsigned int nr) -{ - if (likely(nr == 1)) { - /* - * Optimization: wrprotect_ptes() can only be called for present - * ptes so we only need to check contig bit as condition for - * unfold, and we can remove the contig bit from the pte we read - * to avoid re-reading. This speeds up fork() which is sensitive - * for order-0 folios. Equivalent to contpte_try_unfold(). - */ - pte_t orig_pte = __ptep_get(ptep); - - if (unlikely(pte_cont(orig_pte))) { - __contpte_try_unfold(mm, addr, ptep, orig_pte); - orig_pte = pte_mknoncont(orig_pte); - } - ___ptep_set_wrprotect(mm, addr, ptep, orig_pte); - } else { - contpte_wrprotect_ptes(mm, addr, ptep, nr); - } -} +extern void wrprotect_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, unsigned int nr); #define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - wrprotect_ptes(mm, addr, ptep, 1); -} +extern void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS extern int ptep_set_access_flags(struct vm_area_struct *vma, @@ -1528,7 +1497,8 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_WRPROTECT -#define ptep_set_wrprotect __ptep_set_wrprotect +#define ptep_set_wrprotect(mm, addr, ptep) \ + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)) #define wrprotect_ptes __wrprotect_ptes #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 5675a61452ac..1cef93b15d6e 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -44,21 +44,3 @@ pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); } EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); - -void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr) -{ - /* - * If wrprotecting an entire contig range, we can avoid unfolding. Just - * set wrprotect and wait for the later mmu_gather flush to invalidate - * the tlb. Until the flush, the page may or may not be wrprotected. - * After the flush, it is guaranteed wrprotected. If it's a partial - * range though, we must unfold, because we can't have a case where - * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this - * would cause it to continue to be unpredictable after the flush. - */ - - contpte_try_unfold_partial(mm, addr, ptep, nr); - __wrprotect_ptes(mm, addr, ptep, nr); -} -EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index b151a5aa4de8..728f31da5e6a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -755,11 +755,21 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, } static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) + unsigned long address, pte_t *ptep, + pte_t pte) { atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); } +static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, + pte_t *ptep, unsigned int nr) +{ + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + __ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); +} + static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { @@ -807,6 +817,12 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +extern void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); +extern void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); +#define wrprotect_ptes wrprotect_ptes #else /* CONFIG_THP_CONTPTE */ @@ -822,12 +838,13 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +#define ptep_set_wrprotect(mm, addr, ptep) \ + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)) +#define wrprotect_ptes __wrprotect_ptes #endif /* CONFIG_THP_CONTPTE */ -#define __HAVE_ARCH_PTEP_SET_WRPROTECT -#define ptep_set_wrprotect __ptep_set_wrprotect - #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 76244b0c678a..d1439db1706c 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -26,5 +26,7 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 9cbbff1f67ad..fe36b6b1d20a 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -49,6 +49,8 @@ * - ptep_get_and_clear() * - ptep_test_and_clear_young() * - ptep_clear_flush_young() + * - wrprotect_ptes() + * - ptep_set_wrprotect() */ pte_t huge_ptep_get(pte_t *ptep) @@ -266,7 +268,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t pte; if (!pte_cont(__ptep_get(ptep))) { - __ptep_set_wrprotect(mm, addr, ptep); + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)); return; } @@ -832,4 +834,75 @@ __always_inline int ptep_set_access_flags(struct vm_area_struct *vma, return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); } + +static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * Unfold any partially covered contpte block at the beginning and end + * of the range. + */ + size_t pgsize; + int ncontig; + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + if (ptep != arch_contpte_align_down(ptep) || nr < ncontig) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + if (ptep + nr != arch_contpte_align_down(ptep + nr)) { + unsigned long last_addr = addr + pgsize * (nr - 1); + pte_t *last_ptep = ptep + nr - 1; + + contpte_try_unfold(mm, last_addr, last_ptep, + __ptep_get(last_ptep)); + } +} + +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * If wrprotecting an entire contig range, we can avoid unfolding. Just + * set wrprotect and wait for the later mmu_gather flush to invalidate + * the tlb. Until the flush, the page may or may not be wrprotected. + * After the flush, it is guaranteed wrprotected. If it's a partial + * range though, we must unfold, because we can't have a case where + * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this + * would cause it to continue to be unpredictable after the flush. + */ + + contpte_try_unfold_partial(mm, addr, ptep, nr); + __wrprotect_ptes(mm, addr, ptep, nr); +} +EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); + +__always_inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + if (likely(nr == 1)) { + /* + * Optimization: wrprotect_ptes() can only be called for present + * ptes so we only need to check contig bit as condition for + * unfold, and we can remove the contig bit from the pte we read + * to avoid re-reading. This speeds up fork() which is sensitive + * for order-0 folios. Equivalent to contpte_try_unfold(). + */ + pte_t orig_pte = __ptep_get(ptep); + + if (unlikely(pte_cont(orig_pte))) { + __contpte_try_unfold(mm, addr, ptep, orig_pte); + orig_pte = pte_mknoncont(orig_pte); + } + __ptep_set_wrprotect(mm, addr, ptep, orig_pte); + } else { + contpte_wrprotect_ptes(mm, addr, ptep, nr); + } +} + +__always_inline void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + wrprotect_ptes(mm, addr, ptep, 1); +} #endif /* CONFIG_THP_CONTPTE */