From patchwork Fri Mar 21 13:06:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 14025416 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A95DCC36000 for ; Fri, 21 Mar 2025 13:10:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=+81In4ilDZ/dsxug/tXtQcTDxsQusJyemdqH5+0jxZc=; b=Fmg8CiRX13NzwKGiGFtHuSf41c N7o8FEJkEr70ksjoR7hAHCbu8D/av6aWYqngG5btPxfOTtm49bDeabhQpxHOUeV8ezxpnKsAJQvzX NrB858JfDYeGNhmiM35rdbJvlUxcN1dbQ1zV9thevNuflODI7fcegQR0eZkLg9l/3IRf0VPE9Shw1 XudIaTvX8xhpW58kakRx/oI50Qbep05vSrYJo1WAf/3OPk1LQO1+bWlgMFBR7dOQ1FeM/beCNz9Cr qOH8Zy1xAdXeAWGol6Xka0K1ztAMvqawCPwWJYeRqZlv88MRgHKePhuMXUqcwYLTWmRYRxu179kLX o/HSAohA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvc8e-0000000ErtD-0YW7; Fri, 21 Mar 2025 13:10:16 +0000 Received: from mail-wr1-x432.google.com ([2a00:1450:4864:20::432]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvc69-0000000ErIC-2WMM for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 13:07:42 +0000 Received: by mail-wr1-x432.google.com with SMTP id ffacd0b85a97d-399737f4fa4so1205348f8f.0 for ; Fri, 21 Mar 2025 06:07:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1742562460; x=1743167260; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+81In4ilDZ/dsxug/tXtQcTDxsQusJyemdqH5+0jxZc=; b=HgSJ4whBgtjsbkyb9VF0Ecqve6sdntG50GGllaB71WVRz7zei8i3G/Wx2O/ZduE9Gy cPtm73OyQUvbi0V4k+6VVPNH026CcKEDwsHIiCPaLYzWLY5CBANdBvTe62QWXZMdJeng /nJpWqvB7A31DqHEs9fvHT5WcDouqtZWkwzSvGifw7X10YXri5mx5ncq1g+p2D5jlq5B Y8+kokMnn3jkaCBIOGck4A7FfPAivqbM0PY6oVS/VqeQDrUQQ9x68LIqizI76iykg66h Lb6FLrd/K4+dojG+ceDCrFUTTyLICFzbKmLejoulVYyohVvrmM1oS2q+pOedt1v1RgcL n6Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742562460; x=1743167260; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+81In4ilDZ/dsxug/tXtQcTDxsQusJyemdqH5+0jxZc=; b=R6aRBPUjGKjMLySBqNffK7goiU3TepsIT5xd4ZmL9ouvjWqeXrVM8Dxvzy4298ehum n3rltRW1mcbWKIBrzOp7cbfo8mZdClHxv3V1K/uzxMmjuzR1W+rH9AGLu8zW3CBzYVfo XqDSuy29siSoRftOF9hzdYvpLEfxSdrTG2YCRRDN7bpzrkm/PrazyOGceQYohzXuoFhn ERHuYC3hQWBS2QKVT/s84Y0V7zec4eRgUJoULk2lKmNBvXpjQCv1mN+2yWSL+AM8Rq1L WghOxNIt24jfQlexjR5GpsHiSevgFsSAAfB5A/nbXy8oQjLTq6NUgFvThijE8yfLbH+5 Fn7A== X-Forwarded-Encrypted: i=1; AJvYcCV7HDLM0+CSoQ2ULBiUcpCKLpDDHnmv/x28sWeupFvTlBr2ULXn0ArCuFsMV2SUxMJH7e88PErNt43G2tLwAzAD@lists.infradead.org X-Gm-Message-State: AOJu0Yy3CJMe1Yrc+ead/ElgfxY3FCvQ6iHN5blA2vqh2/4ND1hfqJVv hRnnxMbdSI2M8tP4I7LPdOlLmPRi57zktYhfuANCYbFxv/7eVGlXGY8brrgShBQ= X-Gm-Gg: ASbGncuY2ksmzGC6MfFyDP3vfe3QQ6G+uLhCclF+qnR9KrhHJWVRevHF3ggj/2vIdmV YnK1IVjV2zM6Q29mQykMFvhccs8CkUSqoEOY5ntu2BwA2ONxtY/2qi4Pz3XaFzpMvKO3RRpOcv6 xzyCCT74M9gSor55TO+rto72EfMAU535qAp1zw0DpZj//MOIlIKeQv6xtoZ1ayUAAwa6x6we1Te 4dM9cN3AZXRb7W+YPPArb6TjiMEK4ZaHfk1+jvU8pNZU4jHql8dVJHm0xU/zQhshNPiJuoYq378 hP9g4fpTHd21Hd4lV46O4NYi9Grg9TAq/weFP5uQIubM9Rz4+8BnMSysaLnLOZL8gEU+cA== X-Google-Smtp-Source: AGHT+IFGcNIQFtkSrVrMaEJyHkzdoC3QT/2GtmdRgmqzsJ6C1xedRKK8otT6Nr1+/A9BoBAqiRGLXw== X-Received: by 2002:a05:6000:1f86:b0:391:4743:6de4 with SMTP id ffacd0b85a97d-3997f940e28mr2542677f8f.50.1742562459890; Fri, 21 Mar 2025 06:07:39 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3d12:52fe:c1cc:c94]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3997f9efd6csm2364857f8f.90.2025.03.21.06.07.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Mar 2025 06:07:39 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Matthew Wilcox , Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v5 1/9] riscv: Safely remove huge_pte_offset() when manipulating NAPOT ptes Date: Fri, 21 Mar 2025 14:06:27 +0100 Message-Id: <20250321130635.227011-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250321130635.227011-1-alexghiti@rivosinc.com> References: <20250321130635.227011-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250321_060741_643125_C70F8AC8 X-CRM114-Status: GOOD ( 11.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The pte_t pointer is expected to point to the first entry of the NAPOT mapping so no need to use huge_pte_offset(), similarly to what is done in arm64. Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/hugetlbpage.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 375dd96bb4a0..3192ad804279 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -287,7 +287,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, order = napot_cont_order(pte); pte_num = napot_pte_num(order); - ptep = huge_pte_offset(mm, addr, napot_cont_size(order)); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); if (pte_dirty(orig_pte)) @@ -334,7 +333,6 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, order = napot_cont_order(pte); pte_num = napot_pte_num(order); - ptep = huge_pte_offset(mm, addr, napot_cont_size(order)); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); From patchwork Fri Mar 21 13:06:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 14025427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 088DAC36000 for ; Fri, 21 Mar 2025 13:12:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=MHcTeoTN4G5JoOrCU+WuSXOyCDmNaouJRLSJHK0ciYw=; b=nOOGJphw6VDIyqdjhR5fYKjEHh GqD2ZAV1iWg9Bd0c9aVyySut8yOdtj/rZBLQeP9Q753OU+hhd6TO17gUypGTrU4bi/4Embp2echWn OUvcSPbpzcISsyD7KgM4bkToMBEgNi1ylDOVXyzxo7sU7W6ovzpK4At1zBhX3IrkIGLm9MGrtUdJN Si/GNJHdYGWT/9sIKmgIpspahCGwg/gpd6FyLF+ukxTU3R/sq8nlW6GoNq3pV49hhL/6on6JmHhqs 1ra6zXV+LftLUAxnqk2be4st01D4X5c4uqiNv6JQcabLGjPCyw0R7tpdyLHtpFOyTwtwDZHhBNapi J5PAEcTw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvcAJ-0000000EsPY-09dg; Fri, 21 Mar 2025 13:11:59 +0000 Received: from mail-wm1-x332.google.com ([2a00:1450:4864:20::332]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvc79-0000000ErcE-2v38 for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 13:08:45 +0000 Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-43cfebc343dso14141395e9.2 for ; Fri, 21 Mar 2025 06:08:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1742562522; x=1743167322; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MHcTeoTN4G5JoOrCU+WuSXOyCDmNaouJRLSJHK0ciYw=; b=yxD+T443pJ4HV1SnCCh3624EWLVyFkGWsRO5+qg5lpvNzdl1gKp6m3dwO3JU7RIybl YDirTn1M8pmvfW3CeFRvrOJP3LvcPGx+tD5fzshXcCcpnV1nGg3yGTbgFoDSKLiKjftF uh2tzEsoh4uvElUkdi7Jn/MhNqur9jEP6bFNvta9zD6enAXr1JePuBW2sAoHjDidQrhp I2iz6AJ/CYfNtpF9WobNDfQAvVoLzrmWBK1yzvJ5ZL/uUHz33nuIAj7ldDQaVj0hv4UD YWAAsv3mwb5pz6qEa8V83YgUZzOcPda4+S9GC1geBinHuEW+ZsY+G7pRLs2pIMhBOSdf Ecpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742562522; x=1743167322; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MHcTeoTN4G5JoOrCU+WuSXOyCDmNaouJRLSJHK0ciYw=; b=YJivYfxFkhDP4rN3dJA6GXnoLiGkFlZrVSomPviqtxHQYC76D+jX5NMYONsnGfgZKN DzinLMnMpEiPJ0B4mjEyGAB40InjcwwMfwzzc3kElG8m/u8tEfZS8o9z/wGI8NTZzvTX IzRX+kQiiShVN9nLEu6MjM2AYxvK0Np4LA0PzAj5luralKsVQq3QlrZaK/fzLD47jprk oe0eG3RrP/hKuOpMKeIIOgvet4aKzFZUHwpolzyQ07Kugz4ZS6CIGg29s78q4n364D6i R/jVgc1qDLFhSsHASiD6Xjz5y7SV/4Sk2zqu9dTlY+YMn4L1tKRD4xBX3zWp9fssXbfv PX9g== X-Forwarded-Encrypted: i=1; AJvYcCVOtOpfxE1QsS356dDW5E17YPvaoGSlzK2ee/O6N9a0NT7mJGaBC625TmfoyY7/fsoFP5UpqJdINwKED0+MJs4a@lists.infradead.org X-Gm-Message-State: AOJu0Yw08pms7nSbavA/n9nbhE7QCVSdILfOcp04Eqw6Ya5Mq5jyoiJt QK2XgploPsKnSmYZau5ynlgIozmwKng3W3Tb3abD27T5tCG3zS+y2DjQcKRvFk0= X-Gm-Gg: ASbGncu80noZzQyePXrkzA9UwlWl79oBx0AqIAwWGeo82Cn5EfpCo4sGc43gZXUWrTD 42wQhpiTtXRkcSvCT1M6RvPpDaOpTyyA31VwXXa26YvZufvPfcCxMBJ5FCZEUtfY3sj5XweLfP+ 6XapF48Ri51i6R2Lf2iGFGjq2JcQMtq5SLUrXwi+JbLVmF/zHHIvc7su+7fUtJEkVvOn+16LbNb gghipLAU6vgkwc3VHdEqX2T0yHMTpX7AL3ckDLeQe+L6m87QKQiMwslpcqNn2k9GU7XCMByLvmZ TbSP4lH2gifX+WQFtrWIN6QOAfMKPt2BFvMvlxI6kvwZj9GA29HLd6CcOpIYvQl40uS+37B4zuL vMe5z X-Google-Smtp-Source: AGHT+IEMJBrD5VHSiFntLl7eq4OXY9R2l0VvwmSKG3LcIwl+yD5imfUnEeI+G2u9jHEtl52hrfshfg== X-Received: by 2002:a05:600c:46d4:b0:43c:e6d1:efe7 with SMTP id 5b1f17b1804b1-43d50a3b62emr18553825e9.26.1742562521606; Fri, 21 Mar 2025 06:08:41 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3d12:52fe:c1cc:c94]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-43d43fdec51sm78104015e9.27.2025.03.21.06.08.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Mar 2025 06:08:41 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Matthew Wilcox , Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v5 2/9] riscv: Restore the pfn in a NAPOT pte when manipulated by core mm code Date: Fri, 21 Mar 2025 14:06:28 +0100 Message-Id: <20250321130635.227011-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250321130635.227011-1-alexghiti@rivosinc.com> References: <20250321130635.227011-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250321_060843_748944_65D29023 X-CRM114-Status: GOOD ( 28.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The core mm code expects to be able to extract the pfn from a pte. NAPOT mappings work differently since its ptes actually point to the first pfn of the mapping, the other bits being used to encode the size of the mapping. So modify ptep_get() so that it returns a pte value that contains the *real* pfn (which is then different from what the HW expects) and right before storing the ptes to the page table, reset the pfn LSBs to the size of the mapping. And make sure that all NAPOT mappings are set using set_ptes(). Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/pgtable-64.h | 11 ++ arch/riscv/include/asm/pgtable.h | 155 ++++++++++++++++++++++++---- arch/riscv/mm/hugetlbpage.c | 15 ++- 3 files changed, 152 insertions(+), 29 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 0897dd99ab8d..cddbe426f618 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -104,6 +104,17 @@ enum napot_cont_order { #define napot_cont_mask(order) (~(napot_cont_size(order) - 1UL)) #define napot_pte_num(order) BIT(order) +static inline bool is_napot_order(unsigned int order) +{ + unsigned int napot_order; + + for_each_napot_order(napot_order) + if (order == napot_order) + return true; + + return false; +} + #ifdef CONFIG_RISCV_ISA_SVNAPOT #define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_CONT_ORDER_BASE)) #else diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 050fdc49b5ad..2e62d7e607db 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -296,6 +296,17 @@ static inline unsigned long pte_napot(pte_t pte) return pte_val(pte) & _PAGE_NAPOT; } +#define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) + +/* + * contpte is what we expose to the core mm code, this is not exactly a napot + * mapping since the size is not encoded in the pfn yet. + */ +static inline pte_t pte_mkcont(pte_t pte) +{ + return __pte(pte_val(pte) | _PAGE_NAPOT); +} + static inline pte_t pte_mknapot(pte_t pte, unsigned int order) { int pos = order - 1 + _PAGE_PFN_SHIFT; @@ -305,6 +316,12 @@ static inline pte_t pte_mknapot(pte_t pte, unsigned int order) return __pte((pte_val(pte) & napot_mask) | napot_bit | _PAGE_NAPOT); } +/* pte at entry must *not* encode the mapping size in the pfn LSBs. */ +static inline pte_t pte_clear_napot(pte_t pte) +{ + return __pte(pte_val(pte) & ~_PAGE_NAPOT); +} + #else static __always_inline bool has_svnapot(void) { return false; } @@ -314,17 +331,24 @@ static inline unsigned long pte_napot(pte_t pte) return 0; } +static inline pte_t pte_clear_napot(pte_t pte) +{ + return pte; +} + +static inline pte_t pte_mknapot(pte_t pte, unsigned int order) +{ + return pte; +} + +#define pte_valid_napot(pte) false + #endif /* CONFIG_RISCV_ISA_SVNAPOT */ /* Yields the page frame number (PFN) of a page table entry */ static inline unsigned long pte_pfn(pte_t pte) { - unsigned long res = __page_val_to_pfn(pte_val(pte)); - - if (has_svnapot() && pte_napot(pte)) - res = res & (res - 1UL); - - return res; + return __page_val_to_pfn(pte_val(pte)); } #define pte_page(x) pfn_to_page(pte_pfn(x)) @@ -559,8 +583,13 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT -static inline void set_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pteval, unsigned int nr) +static inline pte_t ___ptep_get(pte_t *ptep) +{ + return READ_ONCE(*ptep); +} + +static inline void ___set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pteval, unsigned int nr) { page_table_check_ptes_set(mm, ptep, pteval, nr); @@ -569,10 +598,13 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; + + if (unlikely(pte_valid_napot(pteval))) + continue; + pte_val(pteval) += 1 << _PAGE_PFN_SHIFT; } } -#define set_ptes set_ptes static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -587,17 +619,6 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addre extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, - unsigned long address, pte_t *ptep) -{ - pte_t pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); - - page_table_check_pte_clear(mm, pte); - - return pte; -} - #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) @@ -627,6 +648,100 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return ptep_test_and_clear_young(vma, address, ptep); } +#ifdef CONFIG_RISCV_ISA_SVNAPOT +static inline pte_t pte_napot_clear_pfn(pte_t *ptep, pte_t pte) +{ + /* + * The pte we load has the N bit set and the size of the mapping in + * the pfn LSBs: keep the N bit and replace the mapping size with + * the *real* pfn since the core mm code expects to find it there. + * The mapping size will be reset just before being written to the + * page table in set_ptes(). + */ + if (unlikely(pte_valid_napot(pte))) { + unsigned int order = napot_cont_order(pte); + int pos = order - 1 + _PAGE_PFN_SHIFT; + unsigned long napot_mask = ~GENMASK(pos, _PAGE_PFN_SHIFT); + pte_t *orig_ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * napot_pte_num(order)); + + pte = __pte((pte_val(pte) & napot_mask) + ((ptep - orig_ptep) << _PAGE_PFN_SHIFT)); + } + + return pte; +} + +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +static inline pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long address, pte_t *ptep) +{ + pte_t pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); + + pte = pte_napot_clear_pfn(ptep, pte); + + page_table_check_pte_clear(mm, pte); + + return pte; +} + +static inline pte_t __ptep_get(pte_t *ptep) +{ + pte_t pte = ___ptep_get(ptep); + + return pte_napot_clear_pfn(ptep, pte); +} + +static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pteval, unsigned int nr) +{ + if (unlikely(pte_valid_napot(pteval))) { + unsigned int order = ilog2(nr); + + if (!is_napot_order(order)) { + /* + * Something's weird, we are given a NAPOT pte but the + * size of the mapping is not a known NAPOT mapping + * size, so clear the NAPOT bit and map this without + * NAPOT support: core mm only manipulates pte with the + * real pfn so we know the pte is valid without the N + * bit. + */ + pr_err("Incorrect NAPOT mapping, resetting.\n"); + pteval = pte_clear_napot(pteval); + } else { + /* + * NAPOT ptes that arrive here only have the N bit set + * and their pfn does not contain the mapping size, so + * set that here. + */ + pteval = pte_mknapot(pteval, order); + } + } + + ___set_ptes(mm, addr, ptep, pteval, nr); +} + +#define ptep_get __ptep_get +#define set_ptes __set_ptes +#define set_contptes(mm, addr, ptep, pte, nr, pgsize) \ + set_ptes(mm, addr, ptep, pte, nr) +#else +#define ptep_get ___ptep_get +#define __ptep_get ___ptep_get +#define set_ptes ___set_ptes +#define __set_ptes ___set_ptes + +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +static inline pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long address, pte_t *ptep) +{ + pte_t pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); + + page_table_check_pte_clear(mm, pte); + + return pte; +} +#endif /* CONFIG_RISCV_ISA_SVNAPOT */ + #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 3192ad804279..60b7e738b31a 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -190,7 +190,7 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) for_each_napot_order(order) { if (shift == napot_cont_shift(order)) { - entry = pte_mknapot(entry, order); + entry = pte_mkcont(entry); break; } } @@ -267,8 +267,7 @@ void set_huge_pte_at(struct mm_struct *mm, clear_flush(mm, addr, ptep, pgsize, pte_num); - for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) - set_pte_at(mm, addr, ptep, pte); + set_ptes(mm, addr, ptep, pte, pte_num); } int huge_ptep_set_access_flags(struct vm_area_struct *vma, @@ -280,7 +279,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; unsigned long order; pte_t orig_pte; - int i, pte_num; + int pte_num; if (!pte_napot(pte)) return ptep_set_access_flags(vma, addr, ptep, pte, dirty); @@ -295,8 +294,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (pte_young(orig_pte)) pte = pte_mkyoung(pte); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) - set_pte_at(mm, addr, ptep, pte); + set_ptes(mm, addr, ptep, pte, pte_num); return true; } @@ -324,7 +322,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t pte = ptep_get(ptep); unsigned long order; pte_t orig_pte; - int i, pte_num; + int pte_num; if (!pte_napot(pte)) { ptep_set_wrprotect(mm, addr, ptep); @@ -337,8 +335,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, orig_pte = pte_wrprotect(orig_pte); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) - set_pte_at(mm, addr, ptep, orig_pte); + set_ptes(mm, addr, ptep, orig_pte, pte_num); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, From patchwork Fri Mar 21 13:06:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 14025428 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 09461C36000 for ; Fri, 21 Mar 2025 13:13:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=e8EtFYpkeeZSqWY8m05eSUUviRVNOc3GBKbkMNHHDeA=; b=AdMZz0b/Ow+DfKfOAkN+uP6ndV QG5IiqDon25D08OOpKAVw0vLpOO+RJdXVFjemnINSr1XrGaAVSawxr4l8VVD3b2BMHYeIyZf6OJaf wTprsk/3d2IK6fDVaryooUm81WUzRhhniRXSyc7gF1QenFWMrU6yrT8ZktWdRwwV/iGd6MQQdkSlx 4nyUfWZldO+mLGjWY5SAh6KknoA0oPx8HHS6jrh5fgKEie1JmOImLQ/Vdw0w34C8Rq24FMAHbpzCo 7PhpabhktjVPmrNiMmKnLcO7uthTu6SVMREwtDZa49D/fizXmgIaIx342QuStTb+tFqGXn3bCISz9 SGT8Q6hQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvcBw-0000000EsjD-3eS3; Fri, 21 Mar 2025 13:13:40 +0000 Received: from mail-wm1-x32a.google.com ([2a00:1450:4864:20::32a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvc88-0000000ErnY-27D9 for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 13:09:45 +0000 Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-43d0359b1fcso12730335e9.0 for ; Fri, 21 Mar 2025 06:09:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1742562583; x=1743167383; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e8EtFYpkeeZSqWY8m05eSUUviRVNOc3GBKbkMNHHDeA=; b=ExBffTHL0Ckf/dH/vmrW5zd+Kybt/YREdyDI348B49LbNFqXuCFpYdQUKNfzkWjfZo 7k/gb0zDZPObw8l8QSPKmaHuig1FcNlTlZngm5gped3hHr/ikU2bT5+G5qlnJZYhVXvn QLeXMKBQ9L0Q8kS8iIVPk4EJfhi+SC0FXaGuvx8GqB+LpZq7owvsIYL0HVKRRAGtsVrY BmZtfloiCHyakYBc2u/5qxfkxyG7N3HTTr8xzE271N0zMCJJwKbuo8IXjQWh2qmiO7AI HFUmcroE634M94GSiTiuTo0WA49ZOwJYe0fhkdtRbeut+BC7MWoAYeGZ74hfBsYp2HHM Ul3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742562583; x=1743167383; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e8EtFYpkeeZSqWY8m05eSUUviRVNOc3GBKbkMNHHDeA=; b=gqoADWIYLJdd6am2CAZMKmAc1S2VLaK5xc2bs0D6vDxp21E5XAiL6UuoNz2YFKLaGt +zb1EjWlaoDUQkkPb7Sf8NkP+WGTqbSpq9LE3jzdfRq9T8PAqCsz1YlNgL6knGc/w73A vChzJ/09zIa4uThUcpXya6d5lTGyAsvkX3pFPQC1WZUkVeJXyCCsALD5Rot3ADudPvuy 2MEuiWuO5rDptz0THGddIe5FnZgzUZa39VA6oVGdiIJGFJdIC4wPnkBLXeLu5tqmO7H/ KCR8+NahN7QDGeaXaq+3hK7mYiOWgqdJAjVI81axqColjXCGvlUlZHfFLbeEfruT9tiA GAzQ== X-Forwarded-Encrypted: i=1; AJvYcCUBcwGCS6Mfe2Uj+g3axZdCUNQadbqqRe0QYjZnPssShMAWNUxgOeX497+OJ1FVegUZiCFhQgDDnoaDg6J+0QWA@lists.infradead.org X-Gm-Message-State: AOJu0Yz7OjcogmSO1QVkvmGc3cy7JPfHlPFXdt47fuyQt97m8uzeUkRV 3KE8KCpaBvG18I3j2gKewe+dRszih7k2LjwGeL/6EFW1UeHKRI6BpG5fyQTKWak= X-Gm-Gg: ASbGnct1luBysOJTikOPeoWq/F22O1DPBX9Aes8dlCg0O9BsFiaPO730hGDDaAPreX/ 8U7cQ5mt9/jOhdwFZ9uENlxD4bxLE9yBlEfXidESvLz1MILx7lJRVUD72+fKyqHcMvz29Xo00w7 CT84F6d2QDfIBE3kZeTK3LIFPXNtt81NfBnn3Tu6F9fcW554qwaNFQWQjgt+NhptkTO5C9RmJpv g2k59L38Nfgm1V/CPTfDFD7gPq8NBA4ymBJJkx5gIjNnNqAQZFbR2+xs+8ywS6nndQGMciVqysb DoKMJ9fPqvRuCGHvQuWs8w46gSUAV2X9hkCYEGAr+2AdndFurk3yACV9kM0Bj4p+rt4a4Q== X-Google-Smtp-Source: AGHT+IFrlDybl2QKCVYqfxTyu64dXX62tEzPiiOD5kkpTcHrUIRBIEyVsyFQKXhsbfpJSLh/l0FWog== X-Received: by 2002:a05:600c:4e4b:b0:439:9737:675b with SMTP id 5b1f17b1804b1-43d502f8536mr33623225e9.7.1742562582938; Fri, 21 Mar 2025 06:09:42 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3d12:52fe:c1cc:c94]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3997f9a3c7fsm2377861f8f.34.2025.03.21.06.09.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Mar 2025 06:09:42 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Matthew Wilcox , Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v5 3/9] mm: Use common huge_ptep_get() function for riscv/arm64 Date: Fri, 21 Mar 2025 14:06:29 +0100 Message-Id: <20250321130635.227011-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250321130635.227011-1-alexghiti@rivosinc.com> References: <20250321130635.227011-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250321_060944_682230_4295B925 X-CRM114-Status: GOOD ( 24.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/hugetlb.h | 3 +- arch/arm64/include/asm/pgtable.h | 57 +++++++++++++++++++++++++-- arch/arm64/mm/hugetlbpage.c | 66 ++------------------------------ arch/riscv/Kconfig | 1 + arch/riscv/include/asm/hugetlb.h | 6 +-- arch/riscv/include/asm/pgtable.h | 45 ++++++++++++++++++++++ arch/riscv/mm/hugetlbpage.c | 62 +++--------------------------- include/linux/hugetlb_contpte.h | 12 ++++++ mm/Kconfig | 3 ++ mm/Makefile | 1 + mm/hugetlb_contpte.c | 32 ++++++++++++++++ 12 files changed, 161 insertions(+), 128 deletions(-) create mode 100644 include/linux/hugetlb_contpte.h create mode 100644 mm/hugetlb_contpte.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 940343beb3d4..5a1e1bc73c15 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -113,6 +113,7 @@ config ARM64 select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_FRAME_POINTERS + select ARCH_WANT_GENERAL_HUGETLB_CONTPTE select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_EXECMEM_LATE diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 07fbf5bf85a7..0604e01dca97 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -13,6 +13,7 @@ #include #include #include +#include #ifdef CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION #define arch_hugetlb_migration_supported arch_hugetlb_migration_supported @@ -53,8 +54,6 @@ extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, #define __HAVE_ARCH_HUGE_PTE_CLEAR extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz); -#define __HAVE_ARCH_HUGE_PTEP_GET -extern pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); void __init arm64_hugetlb_cma_reserve(void); diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 0b2a2ad1b9e8..af8156929c1d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -420,9 +420,10 @@ static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) return pfn_pte(pte_pfn(pte) + nr, pte_pgprot(pte)); } -static inline void __set_ptes(struct mm_struct *mm, - unsigned long __always_unused addr, - pte_t *ptep, pte_t pte, unsigned int nr) +static inline void ___set_ptes(struct mm_struct *mm, + unsigned long __always_unused addr, + pte_t *ptep, pte_t pte, unsigned int nr, + size_t pgsize) { page_table_check_ptes_set(mm, ptep, pte, nr); __sync_cache_and_tags(pte, nr); @@ -433,10 +434,15 @@ static inline void __set_ptes(struct mm_struct *mm, if (--nr == 0) break; ptep++; - pte = pte_advance_pfn(pte, 1); + pte = pte_advance_pfn(pte, pgsize >> PAGE_SHIFT); } } +#define __set_ptes(mm, addr, ptep, pte, nr) \ + ___set_ptes(mm, addr, ptep, pte, nr, PAGE_SIZE) + +#define set_contptes ___set_ptes + /* * Hugetlb definitions. */ @@ -1825,6 +1831,49 @@ static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, #endif /* CONFIG_ARM64_CONTPTE */ +static inline bool __hugetlb_valid_size(unsigned long size) +{ + switch (size) { +#ifndef __PAGETABLE_PMD_FOLDED + case PUD_SIZE: + return pud_sect_supported(); +#endif + case CONT_PMD_SIZE: + case PMD_SIZE: + case CONT_PTE_SIZE: + return true; + } + + return false; +} + +static inline int arch_contpte_get_num_contig(pte_t *ptep, + unsigned long size, + size_t *pgsize) +{ + int contig_ptes = 1; + + if (pgsize) + *pgsize = size; + + switch (size) { + case CONT_PMD_SIZE: + if (pgsize) + *pgsize = PMD_SIZE; + contig_ptes = CONT_PMDS; + break; + case CONT_PTE_SIZE: + if (pgsize) + *pgsize = PAGE_SIZE; + contig_ptes = CONT_PTES; + break; + default: + WARN_ON(!__hugetlb_valid_size(size)); + } + + return contig_ptes; +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index b3a7fafe8892..60a2bb7575c1 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -49,22 +49,6 @@ void __init arm64_hugetlb_cma_reserve(void) } #endif /* CONFIG_CMA */ -static bool __hugetlb_valid_size(unsigned long size) -{ - switch (size) { -#ifndef __PAGETABLE_PMD_FOLDED - case PUD_SIZE: - return pud_sect_supported(); -#endif - case CONT_PMD_SIZE: - case PMD_SIZE: - case CONT_PTE_SIZE: - return true; - } - - return false; -} - #ifdef CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION bool arch_hugetlb_migration_supported(struct hstate *h) { @@ -98,50 +82,6 @@ static int find_num_contig(struct mm_struct *mm, unsigned long addr, return CONT_PTES; } -static inline int num_contig_ptes(unsigned long size, size_t *pgsize) -{ - int contig_ptes = 1; - - *pgsize = size; - - switch (size) { - case CONT_PMD_SIZE: - *pgsize = PMD_SIZE; - contig_ptes = CONT_PMDS; - break; - case CONT_PTE_SIZE: - *pgsize = PAGE_SIZE; - contig_ptes = CONT_PTES; - break; - default: - WARN_ON(!__hugetlb_valid_size(size)); - } - - return contig_ptes; -} - -pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) -{ - int ncontig, i; - size_t pgsize; - pte_t orig_pte = __ptep_get(ptep); - - if (!pte_present(orig_pte) || !pte_cont(orig_pte)) - return orig_pte; - - ncontig = num_contig_ptes(page_size(pte_page(orig_pte)), &pgsize); - for (i = 0; i < ncontig; i++, ptep++) { - pte_t pte = __ptep_get(ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - return orig_pte; -} - /* * Changing some bits of contiguous entries requires us to follow a * Break-Before-Make approach, breaking the whole contiguous set @@ -221,7 +161,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, unsigned long pfn, dpfn; pgprot_t hugeprot; - ncontig = num_contig_ptes(sz, &pgsize); + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) @@ -382,7 +322,7 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, int i, ncontig; size_t pgsize; - ncontig = num_contig_ptes(sz, &pgsize); + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) __pte_clear(mm, addr, ptep); @@ -394,7 +334,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, int ncontig; size_t pgsize; - ncontig = num_contig_ptes(sz, &pgsize); + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); return get_clear_contig(mm, addr, ptep, pgsize, ncontig); } diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 7612c52e9b1e..2a5b2a9f2816 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -79,6 +79,7 @@ config RISCV select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT + select ARCH_WANT_GENERAL_HUGETLB_CONTPTE if RISCV_ISA_SVNAPOT select ARCH_WANT_HUGE_PMD_SHARE if 64BIT select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 446126497768..69393346ade0 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -4,6 +4,9 @@ #include #include +#ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB_CONTPTE +#include +#endif static inline void arch_clear_hugetlb_flags(struct folio *folio) { @@ -44,9 +47,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty); -#define __HAVE_ARCH_HUGE_PTEP_GET -pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 2e62d7e607db..286fe1a32ded 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -296,6 +296,8 @@ static inline unsigned long pte_napot(pte_t pte) return pte_val(pte) & _PAGE_NAPOT; } +#define pte_cont pte_napot + #define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) /* @@ -606,6 +608,49 @@ static inline void ___set_ptes(struct mm_struct *mm, unsigned long addr, } } +#ifdef CONFIG_RISCV_ISA_SVNAPOT +/* + * Some hugetlb functions can be called on !present ptes, so we must use the + * size parameter when it is passed. + */ +static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, + size_t *pgsize) +{ + unsigned long hugepage_shift; + pte_t __pte; + + if (size) { + if (size >= PGDIR_SIZE) + hugepage_shift = PGDIR_SHIFT; + else if (size >= P4D_SIZE) + hugepage_shift = P4D_SHIFT; + else if (size >= PUD_SIZE) + hugepage_shift = PUD_SHIFT; + else if (size >= PMD_SIZE) + hugepage_shift = PMD_SHIFT; + else + hugepage_shift = PAGE_SHIFT; + } else { + /* + * We must read the raw value of the pte to get the size of + * the mapping + */ + __pte = ___ptep_get(ptep); + + /* Make sure __pte is not a swap entry */ + BUG_ON(!pte_valid_napot(__pte)); + + hugepage_shift = PAGE_SHIFT; + size = napot_cont_size(napot_cont_order(__pte)); + } + + if (pgsize) + *pgsize = BIT(hugepage_shift); + + return size >> hugepage_shift; +} +#endif + static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 60b7e738b31a..b9eb6b7b214d 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -3,30 +3,6 @@ #include #ifdef CONFIG_RISCV_ISA_SVNAPOT -pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) -{ - unsigned long pte_num; - int i; - pte_t orig_pte = ptep_get(ptep); - - if (!pte_present(orig_pte) || !pte_napot(orig_pte)) - return orig_pte; - - pte_num = napot_pte_num(napot_cont_order(orig_pte)); - - for (i = 0; i < pte_num; i++, ptep++) { - pte_t pte = ptep_get(ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - - return orig_pte; -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, @@ -215,26 +191,6 @@ static void clear_flush(struct mm_struct *mm, flush_tlb_range(&vma, saddr, addr); } -static int num_contig_ptes_from_size(unsigned long sz, size_t *pgsize) -{ - unsigned long hugepage_shift; - - if (sz >= PGDIR_SIZE) - hugepage_shift = PGDIR_SHIFT; - else if (sz >= P4D_SIZE) - hugepage_shift = P4D_SHIFT; - else if (sz >= PUD_SIZE) - hugepage_shift = PUD_SHIFT; - else if (sz >= PMD_SIZE) - hugepage_shift = PMD_SHIFT; - else - hugepage_shift = PAGE_SHIFT; - - *pgsize = 1 << hugepage_shift; - - return sz >> hugepage_shift; -} - /* * When dealing with NAPOT mappings, the privileged specification indicates that * "if an update needs to be made, the OS generally should first mark all of the @@ -252,7 +208,7 @@ void set_huge_pte_at(struct mm_struct *mm, size_t pgsize; int i, pte_num; - pte_num = num_contig_ptes_from_size(sz, &pgsize); + pte_num = arch_contpte_get_num_contig(ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) @@ -277,15 +233,13 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, int dirty) { struct mm_struct *mm = vma->vm_mm; - unsigned long order; pte_t orig_pte; int pte_num; if (!pte_napot(pte)) return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - order = napot_cont_order(pte); - pte_num = napot_pte_num(order); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); if (pte_dirty(orig_pte)) @@ -303,14 +257,13 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz) { - size_t pgsize; pte_t orig_pte = ptep_get(ptep); int pte_num; if (!pte_napot(orig_pte)) return ptep_get_and_clear(mm, addr, ptep); - pte_num = num_contig_ptes_from_size(sz, &pgsize); + pte_num = arch_contpte_get_num_contig(ptep, sz, NULL); return get_clear_contig(mm, addr, ptep, pte_num); } @@ -320,7 +273,6 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t *ptep) { pte_t pte = ptep_get(ptep); - unsigned long order; pte_t orig_pte; int pte_num; @@ -329,8 +281,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, return; } - order = napot_cont_order(pte); - pte_num = napot_pte_num(order); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); @@ -348,7 +299,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_clear_flush(vma, addr, ptep); - pte_num = napot_pte_num(napot_cont_order(pte)); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } @@ -367,8 +318,7 @@ void huge_pte_clear(struct mm_struct *mm, return; } - pte_num = num_contig_ptes_from_size(sz, &pgsize); - + pte_num = arch_contpte_get_num_contig(ptep, sz, &pgsize); for (i = 0; i < pte_num; i++, addr += pgsize, ptep++) pte_clear(mm, addr, ptep); } diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h new file mode 100644 index 000000000000..2ea17e4fe36b --- /dev/null +++ b/include/linux/hugetlb_contpte.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2025 Rivos Inc. + */ + +#ifndef _LINUX_HUGETLB_CONTPTE_H +#define _LINUX_HUGETLB_CONTPTE_H + +#define __HAVE_ARCH_HUGE_PTEP_GET +extern pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); + +#endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/Kconfig b/mm/Kconfig index 1b501db06417..f9d3f3d49f3e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -853,6 +853,9 @@ config NOMMU_INITIAL_TRIM_EXCESS config ARCH_WANT_GENERAL_HUGETLB bool +config ARCH_WANT_GENERAL_HUGETLB_CONTPTE + bool + config ARCH_WANTS_THP_SWAP def_bool n diff --git a/mm/Makefile b/mm/Makefile index 850386a67b3e..76e8b995f551 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -96,6 +96,7 @@ obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o +obj-$(CONFIG_ARCH_WANT_GENERAL_HUGETLB_CONTPTE) += hugetlb_contpte.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c new file mode 100644 index 000000000000..500d0b96a680 --- /dev/null +++ b/mm/hugetlb_contpte.c @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright 2025 Rivos Inc. + */ + +#include +#include +#include + +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) +{ + int ncontig, i; + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_present(orig_pte) || !pte_cont(orig_pte)) + return orig_pte; + + ncontig = arch_contpte_get_num_contig(ptep, + page_size(pte_page(orig_pte)), + NULL); + + for (i = 0; i < ncontig; i++, ptep++) { + pte_t pte = __ptep_get(ptep); + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + return orig_pte; +} From patchwork Fri Mar 21 13:06:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 14025429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EED8AC36002 for ; Fri, 21 Mar 2025 13:15:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=eAv+4atVMvRknGVbTPj58gq6ySa46Mekau0XoGNQASg=; b=0A4jxvAzG9aOMafgDMX1tPqRuk aERZubQO75YDCBdodq7K0xtlq4Ed49mOl59iDsC1S3/UIE2n7ToUVdR3GESW+/RMQk0WL7HqJlVsr maWWlFs9LvA1SNT2w1jha/+3KxvdoaFogBxm8rW7zwm1x4kh2kDCGlKjBM6Q56m15GO3QzwseeBYx yA0Nb3Ek6+4r69J1OCnHiF+txxxBnyqmlcN16zYxv5mRyLiel0BqVzYuXW1qD6pcvg5jYtBK+hGgi QvGWW5okEt8u9ZmGCKMFnMlH6JGrwV3W4M6NqpqRlVbrHvHukfH829yGNO3gupzTWn64zf5RPh84x lGxGHQXA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvcDc-0000000EtHc-3SeM; Fri, 21 Mar 2025 13:15:24 +0000 Received: from mail-wm1-x333.google.com ([2a00:1450:4864:20::333]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvc98-0000000Es50-1Yhw for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 13:10:48 +0000 Received: by mail-wm1-x333.google.com with SMTP id 5b1f17b1804b1-43d04ea9d9aso9110235e9.3 for ; Fri, 21 Mar 2025 06:10:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1742562645; x=1743167445; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eAv+4atVMvRknGVbTPj58gq6ySa46Mekau0XoGNQASg=; b=mqLhF2BJDjSKTOLH1LlQNhh+5OkssbtWx27he6p+J4ozx+g0Qw4Ht4IvAr0si1Qr51 8w4JSxw1asPouIQDimGyvZHexx9gmQqbcT4yTYzGCEG51yq74/vMRLpNgMmN8j5vsNDq HC4ZPprpUAjDznS7T2bd2H9/CNTVcpJud7/4J/MSXNgbFNeuD6WNKSf7rf7TjpsPeyR1 X1F+6fcgvCed3WzGT7U24cjWyHZK4+KfhfsvHZYCYdagX8K+TSwpnwTAtmKLhj31kBKf 3+UdW8ij84A0hhyR7Iaq7vje7F504/nJg8wdrGbdUYCzKB/v0YtEbOOgxwkgK8rSai3P db9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742562645; x=1743167445; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=eAv+4atVMvRknGVbTPj58gq6ySa46Mekau0XoGNQASg=; b=cKwDHXe2ZfdMZ9FhQcpWv3rOUHvYu/bpHxTOiw6hMmKr+X7UDBf6HhlOhdpWNX+LRH Su7xp7+dLjgKjlxvbANz/b73KeniCNEZ1Y1Gx2aJCbyza9S10eMxs0RzDxLq195W1O8O zzU3y+6FTVKMKGyJoVdyQY0vSkNyDuHoi9qxTGcRjwPSdfSErkpBZgXxMA8FGSFH3C4w Z/UOMakbybF8VrxIOlKsQAd5PH7hvmWptmwUWsUWpCbL6Z1JZE4dEN/JQBLr4cHo/DLP iujlV/Pw6Tzk3NchpYbUEuvrqDpMVY3bvAYnE/xYG9nozbJw6IzXtQ+zbufCFSDRPCYR vMRA== X-Forwarded-Encrypted: i=1; AJvYcCWbpL0svx01Lr5x2ju1v1MoBJXlCupcfjmFHkMn+AUtjDc1AjRJNh3cobmegyrZJLhBe7epZr5Xa772bXP8FUBB@lists.infradead.org X-Gm-Message-State: AOJu0YyIHTN2mnzVoSpmE42R3ejHG0SDVRpyO/FaA5qDnOwvNPTPnGyM XTBTtizyfxNvAD3i1T0fDzYb0yMeXxx8Z9b1XgvrSedtHRUF0HDZQoyJZhYcKPY= X-Gm-Gg: ASbGncul5YnXW+YXAWlHrNs8yWMKiMtYllAgeGkwd92nKZx9NOP5sRuv1FtjN3fP40S tRhSU5l7jiF8DEsux6csv2iSoPNy9NKhMXZbmYH5lFjfIACD9HIXvVmIXxLyZpdW1k8i5sTAe18 zxtkJD2XQwDFnG3BX0tD7QyTNMH1/tHm4z28GGlwQWfJEeYF/SplD+qXtMh56Z/vOxmaei+AQyM m0NauPm4ave1SbmFPo+Mt5kV+SI9bDiEAuO2BvICtbmPtIuuGL37r4b2ORm+F8zAcVkAyUrMDd/ tnVGkfxT0CeMsdmksqJO+5EXZ5rlLLYOWFXr5/RIELkvYGRO7kzVoGqk1b06/P7sFlSMKA== X-Google-Smtp-Source: AGHT+IFFzxLTl7dkoWPQa6CIZER6lpnHzrVNqhIgyBgY3Y3nEtgERu0W1NS2+ffacirYXfcMWjF6NQ== X-Received: by 2002:a05:600c:83cf:b0:439:91dd:cf9c with SMTP id 5b1f17b1804b1-43d509ec70cmr37202275e9.10.1742562644586; Fri, 21 Mar 2025 06:10:44 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3d12:52fe:c1cc:c94]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-43d42b8fcd4sm47982695e9.1.2025.03.21.06.10.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Mar 2025 06:10:44 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Matthew Wilcox , Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v5 4/9] mm: Use common set_huge_pte_at() function for riscv/arm64 Date: Fri, 21 Mar 2025 14:06:30 +0100 Message-Id: <20250321130635.227011-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250321130635.227011-1-alexghiti@rivosinc.com> References: <20250321130635.227011-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250321_061046_406585_2658829B X-CRM114-Status: GOOD ( 20.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 -- arch/arm64/mm/hugetlbpage.c | 56 -------------------------------- arch/riscv/include/asm/hugetlb.h | 5 --- arch/riscv/include/asm/pgtable.h | 13 ++++---- arch/riscv/mm/hugetlbpage.c | 50 ---------------------------- include/linux/hugetlb_contpte.h | 5 +++ mm/hugetlb_contpte.c | 56 ++++++++++++++++++++++++++++++++ 7 files changed, 68 insertions(+), 120 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 0604e01dca97..cfdc04e11585 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -35,9 +35,6 @@ static inline void arch_clear_hugetlb_flags(struct folio *folio) pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT -extern void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned long sz); #define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 60a2bb7575c1..6feb90ed2e7d 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -128,62 +128,6 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm, return orig_pte; } -/* - * Changing some bits of contiguous entries requires us to follow a - * Break-Before-Make approach, breaking the whole contiguous set - * before we can change any entries. See ARM DDI 0487A.k_iss10775, - * "Misprogramming of the Contiguous bit", page D4-1762. - * - * This helper performs the break step for use cases where the - * original pte is not needed. - */ -static void clear_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long i, saddr = addr; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - __ptep_get_and_clear(mm, addr, ptep); - - flush_tlb_range(&vma, saddr, addr); -} - -void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned long sz) -{ - size_t pgsize; - int i; - int ncontig; - unsigned long pfn, dpfn; - pgprot_t hugeprot; - - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); - - if (!pte_present(pte)) { - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - if (!pte_cont(pte)) { - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - pfn = pte_pfn(pte); - dpfn = pgsize >> PAGE_SHIFT; - hugeprot = pte_pgprot(pte); - - clear_flush(mm, addr, ptep, pgsize, ncontig); - - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz) { diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 69393346ade0..7049a17b819d 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -24,11 +24,6 @@ bool arch_hugetlb_migration_supported(struct hstate *h); void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz); -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT -void set_huge_pte_at(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, pte_t pte, - unsigned long sz); - #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 286fe1a32ded..5b34b3c9c0f9 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -715,9 +715,8 @@ static inline pte_t pte_napot_clear_pfn(pte_t *ptep, pte_t pte) return pte; } -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { pte_t pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); @@ -775,9 +774,8 @@ static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, #define set_ptes ___set_ptes #define __set_ptes ___set_ptes -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { pte_t pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); @@ -787,6 +785,9 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, } #endif /* CONFIG_RISCV_ISA_SVNAPOT */ +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +#define ptep_get_and_clear __ptep_get_and_clear + #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index b9eb6b7b214d..75faeacc8138 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -176,56 +176,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -static void clear_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long i, saddr = addr; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - ptep_get_and_clear(mm, addr, ptep); - - flush_tlb_range(&vma, saddr, addr); -} - -/* - * When dealing with NAPOT mappings, the privileged specification indicates that - * "if an update needs to be made, the OS generally should first mark all of the - * PTEs invalid, then issue SFENCE.VMA instruction(s) covering all 4 KiB regions - * within the range, [...] then update the PTE(s), as described in Section - * 4.2.1.". That's the equivalent of the Break-Before-Make approach used by - * arm64. - */ -void set_huge_pte_at(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - pte_t pte, - unsigned long sz) -{ - size_t pgsize; - int i, pte_num; - - pte_num = arch_contpte_get_num_contig(ptep, sz, &pgsize); - - if (!pte_present(pte)) { - for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) - set_ptes(mm, addr, ptep, pte, 1); - return; - } - - if (!pte_napot(pte)) { - set_ptes(mm, addr, ptep, pte, 1); - return; - } - - clear_flush(mm, addr, ptep, pgsize, pte_num); - - set_ptes(mm, addr, ptep, pte, pte_num); -} - int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index 2ea17e4fe36b..135b68bd09ca 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -9,4 +9,9 @@ #define __HAVE_ARCH_HUGE_PTEP_GET extern pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +extern void set_huge_pte_at(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, pte_t pte, + unsigned long sz); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index 500d0b96a680..cbf93ffcd882 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -30,3 +30,59 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) } return orig_pte; } + +/* + * ARM64: Changing some bits of contiguous entries requires us to follow a + * Break-Before-Make approach, breaking the whole contiguous set + * before we can change any entries. See ARM DDI 0487A.k_iss10775, + * "Misprogramming of the Contiguous bit", page D4-1762. + * + * RISCV: When dealing with NAPOT mappings, the privileged specification + * indicates that "if an update needs to be made, the OS generally should first + * mark all of the PTEs invalid, then issue SFENCE.VMA instruction(s) covering + * all 4 KiB regions within the range, [...] then update the PTE(s), as + * described in Section 4.2.1.". That's the equivalent of the Break-Before-Make + * approach used by arm64. + * + * This helper performs the break step for use cases where the + * original pte is not needed. + */ +static void clear_flush(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long i, saddr = addr; + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) + __ptep_get_and_clear(mm, addr, ptep); + + flush_tlb_range(&vma, saddr, addr); +} + +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned long sz) +{ + size_t pgsize; + int i; + int ncontig; + + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + + if (!pte_present(pte)) { + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + __set_ptes(mm, addr, ptep, pte, 1); + return; + } + + if (!pte_cont(pte)) { + __set_ptes(mm, addr, ptep, pte, 1); + return; + } + + clear_flush(mm, addr, ptep, pgsize, ncontig); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); +} From patchwork Fri Mar 21 13:06:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 14025438 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 17EE1C36001 for ; Fri, 21 Mar 2025 13:17:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=g+Ba9lH6i+4JHX31VX+vvn1rf15Jq1WxTsiq92NtkJo=; b=tlhedr73MzPD9y6fiFgcQjrZ+X ri5E1ycws+wF9VyhEmkCavhPmcBEjcN+w3H2BNYTeVLJy0D5meWdvcdrrHYOomNUIYEcEGdaNKcYP H30QoirJR4mqjeds96sac3jZNyTxEK5gT7kDAG4KvWs3VCda+6tSfsp4S9AmKjkzAtjpi+8hqyhUN 56AHADdNaPq7xjl9b2cPh0fKqdykdQbzZDXi7cYGKTD4SsjQvdVq12ULwZL9A/VQqwT16Qjv1HVuP dfkdJ/B9peBzFza5ipg8M+WNcKDcAFAGyfvQX+wu5OA+7gbkAsCvAi0M1o5CNGx2htqD7B+9CPkbi epUU7h0A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvcFG-0000000Etbb-2NBB; Fri, 21 Mar 2025 13:17:06 +0000 Received: from mail-wr1-x42d.google.com ([2a00:1450:4864:20::42d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvcA7-0000000EsLp-2MTy for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 13:11:49 +0000 Received: by mail-wr1-x42d.google.com with SMTP id ffacd0b85a97d-39141ffa9fcso1751276f8f.0 for ; Fri, 21 Mar 2025 06:11:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1742562706; x=1743167506; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g+Ba9lH6i+4JHX31VX+vvn1rf15Jq1WxTsiq92NtkJo=; b=gAT+RXyYDzHUhNh+se7154L412bJnmpvmO7n8rDmf2+H3R5GQTuccUUmYnnZYWZsfI TNRZRO4G/zVSuaTjJDour7qFUaQLGnC8QnwU+J398Knw3fZTdbs/tvBrLMNwo5QX8dX6 0NSyekD0p55jyuPsSDjzY3ql1jDPRgDrhCj2+G+RIqBAramF2e7Dkz2XKBjooEzxSUDM AbOXxbExRdr1/aTTbrsaABNH9lEUvjjqSU1m2U7AZ+Mthy9hQViE7zxFW7wyeoYFnbpB c7OI12TC/DEGV9sqEd3bPHPjqEagfq0/SDc6mtlLdKjbASDKKRVA37PcdSnIykW6rqWf 5btg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742562706; x=1743167506; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g+Ba9lH6i+4JHX31VX+vvn1rf15Jq1WxTsiq92NtkJo=; b=GxW8woXoZhZKUKmnFnbAZPUU3Qi2hhgLOAD6mSiZkZFoXRTYiO9okypFILip5J0cXG eBJcohQUYzlsAK4MGdaDhQ6ZMiFt9KLjd/IVFhpaK+FmG/+vpQyxB7MFzBWzC6IWyPA1 Bu99shGFDb5pMOaTYBcE10gwQAPHplv1/XQRdPw6l0CYZkBlzy2msqOXv58+FwYO5D4t J9hqS+E9nfI6fDaIQmCbizCCQ6VI0UIqzYobrqrrdjscDzVYI9aF9lVA1nGztIWyReCP iytArWKsa+6Uyp46l0WdGjV4N+0e7GLwGkyHqt2tz90kF29gAMxGi7saMO1OfL7qgqT5 igEA== X-Forwarded-Encrypted: i=1; AJvYcCXnHfllkfBqWDDoAANMl4VhVedexqP4ZHsqBARDayFG81Ytrg+xXhmwnyJIk8+02pq74W/FBdza1qoxUgorDo2U@lists.infradead.org X-Gm-Message-State: AOJu0YzfebueUODxRhGAcLvxDiDjLAmuBYdlt1duYXBgBKkk3RwMEnZ2 /5R7GzztQMypPljk/IFgFd02pP3FiOVT8My+8WFZIHM8gp9JXjvFefpG2C93Oqk= X-Gm-Gg: ASbGnctmUAURpevcV6dw3nNzbri4p9HiIZhntKSyWsDqbjvcVCJ7ByZzJcErsc+t3lI szJnQ2r7LZfIziclbHpw0wK1dbkmosrgHlTMGmmw7Kevkb8C3mwxi7rkidVyf2YL/vvTKf+1fO5 heRP7fDvalie7k3BYfJiJUWAfKzsWKHskp+urrCwHPcfExMRhvnB3JfCjJ/j7Wn62zwgzfaUF1a nv78UGdatwnj9VqiDk5qFz3kOH4FLmZRiwI+7EPk+uuVMATl9rxX6u7VkujxXt58UzqMRVm1mMX aPD5zkA8sjRs3j/rD7Zr7M3TjC8AIwB5jN4cMIWfHxSb7EBJXjnuSpdI1uNyQa4+VF2tsA== X-Google-Smtp-Source: AGHT+IEGZPDoYAPQGYmkPYx5aU5RVlINcZRG7cNrGB3gR0sMYyiWL8XPHTT0J17da9GbtboJpKfvoQ== X-Received: by 2002:a05:6000:154a:b0:38f:3224:660b with SMTP id ffacd0b85a97d-3997f90a69bmr2874098f8f.22.1742562705888; Fri, 21 Mar 2025 06:11:45 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3d12:52fe:c1cc:c94]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3997f9b26a6sm2310275f8f.44.2025.03.21.06.11.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Mar 2025 06:11:45 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Matthew Wilcox , Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v5 5/9] mm: Use common huge_pte_clear() function for riscv/arm64 Date: Fri, 21 Mar 2025 14:06:31 +0100 Message-Id: <20250321130635.227011-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250321130635.227011-1-alexghiti@rivosinc.com> References: <20250321130635.227011-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250321_061147_604907_9307B347 X-CRM114-Status: GOOD ( 14.37 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Both architectures have the same implementation so move it to generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 --- arch/arm64/mm/hugetlbpage.c | 12 ------------ arch/riscv/include/asm/hugetlb.h | 4 ---- arch/riscv/include/asm/pgtable.h | 5 +++-- arch/riscv/mm/hugetlbpage.c | 19 ------------------- include/linux/hugetlb_contpte.h | 4 ++++ mm/hugetlb_contpte.c | 12 ++++++++++++ 7 files changed, 19 insertions(+), 40 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index cfdc04e11585..ed75631ad63c 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -48,9 +48,6 @@ extern void huge_ptep_set_wrprotect(struct mm_struct *mm, #define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); -#define __HAVE_ARCH_HUGE_PTE_CLEAR -extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long sz); void __init arm64_hugetlb_cma_reserve(void); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 6feb90ed2e7d..99728b02a3ca 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -260,18 +260,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -void huge_pte_clear(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long sz) -{ - int i, ncontig; - size_t pgsize; - - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - __pte_clear(mm, addr, ptep); -} - pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz) { diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 7049a17b819d..467bc30c2153 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -20,10 +20,6 @@ bool arch_hugetlb_migration_supported(struct hstate *h); #endif #ifdef CONFIG_RISCV_ISA_SVNAPOT -#define __HAVE_ARCH_HUGE_PTE_CLEAR -void huge_pte_clear(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long sz); - #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 5b34b3c9c0f9..72d3592454d3 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -651,8 +651,8 @@ static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, } #endif -static inline void pte_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) +static inline void __pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) { __set_pte_at(mm, ptep, __pte(0)); } @@ -787,6 +787,7 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear +#define pte_clear __pte_clear #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 75faeacc8138..fe82284c3dc4 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -254,25 +254,6 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } -void huge_pte_clear(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long sz) -{ - size_t pgsize; - pte_t pte = ptep_get(ptep); - int i, pte_num; - - if (!pte_napot(pte)) { - pte_clear(mm, addr, ptep); - return; - } - - pte_num = arch_contpte_get_num_contig(ptep, sz, &pgsize); - for (i = 0; i < pte_num; i++, addr += pgsize, ptep++) - pte_clear(mm, addr, ptep); -} - static bool is_napot_size(unsigned long size) { unsigned long order; diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index 135b68bd09ca..e6aa9befa78c 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -14,4 +14,8 @@ extern void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned long sz); +#define __HAVE_ARCH_HUGE_PTE_CLEAR +extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned long sz); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index cbf93ffcd882..e881b302dd63 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -86,3 +86,15 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, set_contptes(mm, addr, ptep, pte, ncontig, pgsize); } + +void huge_pte_clear(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned long sz) +{ + int i, ncontig; + size_t pgsize; + + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) + __pte_clear(mm, addr, ptep); +} From patchwork Fri Mar 21 13:06:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 14025448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E4C76C36000 for ; Fri, 21 Mar 2025 13:26:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=tW5CZGKlc6dgOhT2HYLkHYDVp73Jiwu1Hc2Rif5lDWM=; b=ODc6NdJBjSTW6SCqhlGcYrfI0N WRsRejp35blMJnngQepp9vjJ3dQY1LS2KaGYVvH8V+aVKzbAkxh1erW64bHqXrcHLBTjp8HtO8ONP 2r2Bw5w3XvEr96k49X70487pLGPPzJJTt5DJR2uBZ/oRymf03tI/L18NWYvx2tdQp8ykxYwxaFucb DLuXe/sJFfh74vC71PqeQV02SKrg+qQB3zNFlvLdfxmMV2Fhs7YrPv39+9C95ZNoO/nHnZKzWfYaM PHygT0G4Qii5hST9d5gsoJfqm19L/pIyCMUIVtJIV5XHXv5lBw0cqSKNW2iVzk5+YEX73hZUP1DXj 3pBO8ogA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvcNv-0000000EvJQ-0ElF; Fri, 21 Mar 2025 13:26:03 +0000 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvcB7-0000000EsYy-1EUn for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 13:12:51 +0000 Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-3914bc3e01aso1326735f8f.2 for ; Fri, 21 Mar 2025 06:12:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1742562768; x=1743167568; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tW5CZGKlc6dgOhT2HYLkHYDVp73Jiwu1Hc2Rif5lDWM=; b=puYwddz8Wz+5V5YrPz2C2AcTgvDT4mKtmkEeH4loATwSKRXgec/hBnnIHbVyY5iX99 XJbj7o+XUQz9QwtTeXsxV1+T95GhTQaVmC+gfCjXLN0ODUSMwH3joBtcrm9ZeCbgMy5l Ek/JROE7oNeRHAtrUxE/D2hTQBHy3trPvPcL6YHFW+l5uLcwW4AswEzsCyDJ8C2LB9MC QzrCr6oOEGQcE2oFvjVSGLuk19+uwLUn0g6zeszYFCUM02Vqwt3vJv6lkLR9VXMzXahT nS3aZhotDxEDXVdgx0M0YZ/wopd1FURCesZiNGP3gO364lvPf2zs4+COlqz7eUriMq+1 DyAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742562768; x=1743167568; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tW5CZGKlc6dgOhT2HYLkHYDVp73Jiwu1Hc2Rif5lDWM=; b=WjWT+B4KDqzKoR5MznvhdEpROrKdJu72Uwbt5tpS5gtvNE76dfF8n7nCs94fZSxSGk HEhyXkBshxPbIt0AGYBHoF8p68PC0ETdodo1UEjkIL6wRgIweCWpooKaKZLBIWghWwlL PddPLM3l6xsUIY8byhnF4AWsJ/ouYAFYbZsu1fyFG5z1MKg/QnAQ73YiHvmcQAkA4PAJ AQcRhcYjrmakdRpxvSrSSaRXVsgraYxT5SuqoU4vYOlKobG1GGZJ+YRMblsgVDObQK5Y C9qQVHvGWB4u3w7iFtoSSpY8WRFey9Etl5JBrjQRMYqTyutbq7y8VqT39feSCg6NBCBy JNoQ== X-Forwarded-Encrypted: i=1; AJvYcCV8B40TK2217SeO2veEf4ELPV/XRYiKssc9jQHHX13g3wIdUL30gpD3iLJFq/vW9PT91Hwcj7mp/VHnriuCqW4p@lists.infradead.org X-Gm-Message-State: AOJu0YxNawQdn23FyYOL/qQujt2T10RvvMerk5+t+njJFe4r5+uA6Tex mCO41haqGd3YNwAQakBhi8xXKn9H9bWCpwN60w0PAnLxUqaaeD4JuxN8frtDKgk= X-Gm-Gg: ASbGnct3UTgz03YIZXC3i/M7TqYg6zuqERa5fSIpkx/qV6d5Y1RfwaPjPjskCazItj1 BOP6AnbfU04rz/ShTeT5eGKqq/DBEvnz3SqMjLeC/+1qr5k9agtjI5DiX3vDeK8tr+XKlajkfDj DWwRGxIS+gMirtHwxEtaYTjomKL1doqWAaYATkYxzCJGgOCvCqIiiuJHRHPvaJy/4xbON6qtk1O 5m7dgaiUoEz9pQt3ceWC7MTnUOr/jJ98DY8DKO+ID2RwJir5GCNsPqUsFiTF5MVlZIGlVPNplo3 CwZ1x70LyuuSefEKxmIFL33kcdHVKxyOnH6thTQRjmlVezi5EQFuvz9r3WosQ7Rprbdwcg== X-Google-Smtp-Source: AGHT+IEUtROIBbu/cy/pImYWjl+q4lRngYo9z7idT6KT9+6QFPXCkP6BZJtyCAPZK3CSSAem4Gc0LA== X-Received: by 2002:a5d:64af:0:b0:391:2db0:2961 with SMTP id ffacd0b85a97d-3997f932efcmr3394631f8f.38.1742562767493; Fri, 21 Mar 2025 06:12:47 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3d12:52fe:c1cc:c94]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3997f9e66c7sm2356167f8f.79.2025.03.21.06.12.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Mar 2025 06:12:47 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Matthew Wilcox , Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v5 6/9] mm: Use common huge_ptep_get_and_clear() function for riscv/arm64 Date: Fri, 21 Mar 2025 14:06:32 +0100 Message-Id: <20250321130635.227011-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250321130635.227011-1-alexghiti@rivosinc.com> References: <20250321130635.227011-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250321_061249_337992_DAAD15EB X-CRM114-Status: GOOD ( 18.24 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Note that get_clear_contig() function is duplicated in the generic and the arm64 code because it is still used by some arm64 functions that will, in the next commits, be moved to the generic code. Once all have been moved, the arm64 version will be removed. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 -- arch/arm64/mm/hugetlbpage.c | 10 ------- arch/riscv/include/asm/hugetlb.h | 5 ---- arch/riscv/mm/hugetlbpage.c | 15 ---------- include/linux/hugetlb_contpte.h | 5 ++++ mm/hugetlb_contpte.c | 51 ++++++++++++++++++++++++++++++++ 6 files changed, 56 insertions(+), 33 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index ed75631ad63c..9b1c25775bea 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -39,9 +39,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty); -#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR -extern pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long sz); #define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT extern void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 99728b02a3ca..62a66ce2b2fe 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -260,16 +260,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long sz) -{ - int ncontig; - size_t pgsize; - - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); - return get_clear_contig(mm, addr, ptep, pgsize, ncontig); -} - /* * huge_ptep_set_access_flags will update access flags (dirty, accesssed) * and write permission. diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 467bc30c2153..0fbb6b19df79 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -20,11 +20,6 @@ bool arch_hugetlb_migration_supported(struct hstate *h); #endif #ifdef CONFIG_RISCV_ISA_SVNAPOT -#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, - unsigned long sz); - #define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index fe82284c3dc4..87168123d4a2 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -203,21 +203,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return true; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, unsigned long sz) -{ - pte_t orig_pte = ptep_get(ptep); - int pte_num; - - if (!pte_napot(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); - - pte_num = arch_contpte_get_num_contig(ptep, sz, NULL); - - return get_clear_contig(mm, addr, ptep, pte_num); -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index e6aa9befa78c..1c8f46ff95ea 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -18,4 +18,9 @@ extern void set_huge_pte_at(struct mm_struct *mm, extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz); +#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR +extern pte_t huge_ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned long sz); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index e881b302dd63..82f49eb79ffb 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -98,3 +98,54 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) __pte_clear(mm, addr, ptep); } + +/* + * ARM: Changing some bits of contiguous entries requires us to follow a + * Break-Before-Make approach, breaking the whole contiguous set + * before we can change any entries. See ARM DDI 0487A.k_iss10775, + * "Misprogramming of the Contiguous bit", page D4-1762. + * + * RISCV: When dealing with NAPOT mappings, the privileged specification + * indicates that "if an update needs to be made, the OS generally should first + * mark all of the PTEs invalid, then issue SFENCE.VMA instruction(s) covering + * all 4 KiB regions within the range, [...] then update the PTE(s), as + * described in Section 4.2.1.". That's the equivalent of the Break-Before-Make + * approach used by arm64. + * + * This helper performs the break step. + */ +static pte_t get_clear_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + pte_t pte, tmp_pte; + bool present; + + pte = __ptep_get_and_clear(mm, addr, ptep); + present = pte_present(pte); + while (--ncontig) { + ptep++; + addr += pgsize; + tmp_pte = __ptep_get_and_clear(mm, addr, ptep); + if (present) { + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + } + return pte; +} + +pte_t huge_ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, unsigned long sz) +{ + int ncontig; + size_t pgsize; + + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + + return get_clear_contig(mm, addr, ptep, pgsize, ncontig); +} From patchwork Fri Mar 21 13:06:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 14025439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F2674C36000 for ; Fri, 21 Mar 2025 13:19:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ws5N2/PtP1Eu6ihB8rghsKDx2GXJblSk7INWyBMy7EM=; b=Sy/uFC7z0+WODLHYMAIQIOj/45 GHtf8Riwo/EIHeIaU0qt78XU1K2LQ+/22tJVG3q+TjxW7q0ac37wuVLBwZpnJaxpPEtpA/JQBgv0x NSB3XLetY1B+UM9uL3TmsJirutTVDZL3sR+coLhLZoAPe2TYEMIMLGM9VxCrBXim202wVME1HP+O9 aR0SICUYH3rvTn24laPfvtAY9AQ3UHW6DeHIAwjcuwmJ5nPCLSplIPDdXjV7gKA3husqbnaiwceUG J3tu+sE4qn/AlCWPFEz1GYjILqTvtt90hesSV3B8y7jnRXm5hyZbIAUv+43+ZI98gVExxLrVgo3Dp oAolXGDA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvcHI-0000000Eu8N-2kZI; Fri, 21 Mar 2025 13:19:12 +0000 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvcC6-0000000EsoK-1tRL for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 13:13:51 +0000 Received: by mail-wr1-x433.google.com with SMTP id ffacd0b85a97d-3913fdd0120so1169120f8f.0 for ; Fri, 21 Mar 2025 06:13:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1742562829; x=1743167629; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ws5N2/PtP1Eu6ihB8rghsKDx2GXJblSk7INWyBMy7EM=; b=rGxdA2iSYINKPn3cO39fQoRkfxpSAKZAWuKlPU2M+ygjx6Sx9ia+nEufNLfsyzfVzk uWLvxuuxZYf+Fh1Px+ZsxdyU1rO9ADn+ZUi3Y2mcbFSlu/w6CqPJYunnugDtVfkruEBv GJWkY8aOU5xfQPkZUNf965sLGzbH1WagyUpUL+Mb3pJDjTp7jDYuMjibQlHvvDFQrGEP VPFSuDgxGFe+2xo1eNSbvLCPuwsdHQcQrfZA37sDGR2+iwb6+EYb4kK21uGftxtF7JdR KzvfrzXRxDqlPIDLcTEgrct+Tq8UyEglejzBV4XoL0wZbz2tshj0P5rkBVXthffZtZ8/ kU7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742562829; x=1743167629; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ws5N2/PtP1Eu6ihB8rghsKDx2GXJblSk7INWyBMy7EM=; b=CLxO13EWE5B6cMrqs0SA0Ii+1IoZodza6r6N2YvPnavFwGexufcJpW4hPRFiTrIwS0 Uj+6qLYqMitkstcwIZXoAqJZjnK43R6ILjVubVGTDuL3/ZRXjm8sc/U6Ub/liRAzjYMa W8MHZL5PGmtZkRvJspa28ZjADeU2dBOeE0HHreETE3l3sm1w0mGPZ7AlkxmlLhnpBxMl J3tdixmr1tGVwTfLJbZWwC6gBPPk4ytdnMpidM7OOmasNDeebKRi7A8UsRFRI4F1BhXh 33bMRZLyj1HQvRWlWGvof3LjKghgehXJm59E8amhaA+60Ajeb54HSdTSR0pXYBXn/rdL 7y1w== X-Forwarded-Encrypted: i=1; AJvYcCX95BxQi1ZAulk6XmBS3v+JzcUy0ExE8PjF+LDsMI82MW37WVyMu9uRK52g53ZvAws+OWqa6ijpdB14kC5bkNQW@lists.infradead.org X-Gm-Message-State: AOJu0Yx+kaWucuyCzxFa/q3BxHtw+c+AJIYL7W3bgHZInY2Qk0g5rDeZ YOM+4Yx7Mpfv4pWDoBD0R0VGO1X/zHYo0aoueJl34jNwV5Gqr2rfV+1fuVLiHeI= X-Gm-Gg: ASbGncsLu2n5UmmIdMuS9rzudSYIvX+J/h/Lu7JAgJsa1Q0hX3ES2lk9iKmtWoYkd59 5pATYFh8THpwQiFmK4yNtZG1qSFLp1jizcqm7TIsoqh1b8g3uzWw22wnkeQdzrB1a5CwwgmT+pJ DIQzPjd0beagDW2uqPC1PsHNBf3pAfiI4A/XLtLt2S2lQVhDVO3GmuCsWpVsVAegjESE3Em6mlV dWFjwOcUoYtuArLYhmPk0c5jg1ZiAIRQSlpkx31unXAVm7rsWbqaQObnZoOJreUehrZc+Wn1ZL8 BiZ5mw60pq3jTzY0fEtRKeoUqp5vSMq6DZZN7n64+kf9YHxTTnDFRVTHjLgbQ6+i0l3qP+M16YF gBdim X-Google-Smtp-Source: AGHT+IGUJnxhjyGu7ufB6wyGpdYdbKS7+6k2eyHYgFKxVHRlB/Xm7E8vrXAFhp0hvlUzI9R7KED60g== X-Received: by 2002:a5d:64af:0:b0:391:2db0:2961 with SMTP id ffacd0b85a97d-3997f932efcmr3397837f8f.38.1742562828869; Fri, 21 Mar 2025 06:13:48 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3d12:52fe:c1cc:c94]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3997f9e674fsm2292970f8f.80.2025.03.21.06.13.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Mar 2025 06:13:48 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Matthew Wilcox , Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v5 7/9] mm: Use common huge_ptep_set_access_flags() function for riscv/arm64 Date: Fri, 21 Mar 2025 14:06:33 +0100 Message-Id: <20250321130635.227011-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250321130635.227011-1-alexghiti@rivosinc.com> References: <20250321130635.227011-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250321_061350_617243_78F7C507 X-CRM114-Status: GOOD ( 25.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Both architectures have almost the same implementation: __cont_access_flags_changed() is also correct on riscv and brings the same benefits (ie don't do anything if the flags are unchanged). As in the previous commit, get_clear_contig_flush() is duplicated in both the arch and the generic codes, it will be removed from the arch code when the last reference there gets moved to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 4 -- arch/arm64/include/asm/pgtable.h | 15 +++++- arch/arm64/mm/hugetlbpage.c | 69 +-------------------------- arch/riscv/include/asm/hugetlb.h | 5 -- arch/riscv/include/asm/pgtable.h | 11 +++-- arch/riscv/mm/hugetlbpage.c | 32 ++----------- arch/riscv/mm/pgtable.c | 6 +-- include/linux/hugetlb_contpte.h | 5 ++ mm/hugetlb_contpte.c | 81 ++++++++++++++++++++++++++++++-- 9 files changed, 110 insertions(+), 118 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 9b1c25775bea..29a9dac52cef 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -35,10 +35,6 @@ static inline void arch_clear_hugetlb_flags(struct folio *folio) pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS -extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t pte, int dirty); #define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT extern void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep); diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index af8156929c1d..9b5c57e56691 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1847,12 +1847,23 @@ static inline bool __hugetlb_valid_size(unsigned long size) return false; } -static inline int arch_contpte_get_num_contig(pte_t *ptep, - unsigned long size, +extern int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize); + +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { int contig_ptes = 1; + /* + * If the size is not passed, we need to go through the page table to + * find out the number of contiguous ptes. + */ + if (size == 0) + return find_num_contig(mm, addr, ptep, pgsize); + if (pgsize) *pgsize = size; diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 62a66ce2b2fe..03cb757f7935 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -63,8 +63,8 @@ bool arch_hugetlb_migration_supported(struct hstate *h) } #endif -static int find_num_contig(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, size_t *pgsize) +int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize) { pgd_t *pgdp = pgd_offset(mm, addr); p4d_t *p4dp; @@ -260,71 +260,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -/* - * huge_ptep_set_access_flags will update access flags (dirty, accesssed) - * and write permission. - * - * For a contiguous huge pte range we need to check whether or not write - * permission has to change only on the first pte in the set. Then for - * all the contiguous ptes we need to check whether or not there is a - * discrepancy between dirty or young. - */ -static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig) -{ - int i; - - if (pte_write(pte) != pte_write(__ptep_get(ptep))) - return 1; - - for (i = 0; i < ncontig; i++) { - pte_t orig_pte = __ptep_get(ptep + i); - - if (pte_dirty(pte) != pte_dirty(orig_pte)) - return 1; - - if (pte_young(pte) != pte_young(orig_pte)) - return 1; - } - - return 0; -} - -int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t pte, int dirty) -{ - int ncontig, i; - size_t pgsize = 0; - unsigned long pfn = pte_pfn(pte), dpfn; - struct mm_struct *mm = vma->vm_mm; - pgprot_t hugeprot; - pte_t orig_pte; - - if (!pte_cont(pte)) - return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - dpfn = pgsize >> PAGE_SHIFT; - - if (!__cont_access_flags_changed(ptep, pte, ncontig)) - return 0; - - orig_pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); - - /* Make sure we don't lose the dirty or young state */ - if (pte_dirty(orig_pte)) - pte = pte_mkdirty(pte); - - if (pte_young(orig_pte)) - pte = pte_mkyoung(pte); - - hugeprot = pte_pgprot(pte); - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); - - return 1; -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 0fbb6b19df79..bf533c2cef84 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -28,11 +28,6 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep); -#define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS -int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t pte, int dirty); - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 72d3592454d3..081385e0d10a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -613,7 +613,9 @@ static inline void ___set_ptes(struct mm_struct *mm, unsigned long addr, * Some hugetlb functions can be called on !present ptes, so we must use the * size parameter when it is passed. */ -static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { unsigned long hugepage_shift; @@ -657,9 +659,8 @@ static inline void __pte_clear(struct mm_struct *mm, __set_pte_at(mm, ptep, __pte(0)); } -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS /* defined in mm/pgtable.c */ -extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, - pte_t *ptep, pte_t entry, int dirty); +extern int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, + pte_t *ptep, pte_t entry, int dirty); #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); @@ -788,6 +789,8 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear #define pte_clear __pte_clear +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +#define ptep_set_access_flags __ptep_set_access_flags #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 87168123d4a2..b2046f4bd445 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -176,33 +176,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, - pte_t *ptep, - pte_t pte, - int dirty) -{ - struct mm_struct *mm = vma->vm_mm; - pte_t orig_pte; - int pte_num; - - if (!pte_napot(pte)) - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); - orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); - - if (pte_dirty(orig_pte)) - pte = pte_mkdirty(pte); - - if (pte_young(orig_pte)) - pte = pte_mkyoung(pte); - - set_ptes(mm, addr, ptep, pte, pte_num); - - return true; -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -216,7 +189,8 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, return; } - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); + pte_num = arch_contpte_get_num_contig(mm, addr, ptep, 0, NULL); + orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); @@ -234,7 +208,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_clear_flush(vma, addr, ptep); - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 4ae67324f992..af8b3769a349 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -5,9 +5,9 @@ #include #include -int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, - pte_t entry, int dirty) +int __ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty) { asm goto(ALTERNATIVE("nop", "j %l[svvptc]", 0, RISCV_ISA_EXT_SVVPTC, 1) : : : : svvptc); diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index 1c8f46ff95ea..e129578f6500 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -23,4 +23,9 @@ extern pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz); +#define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS +extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, int dirty); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index 82f49eb79ffb..b4c409d11195 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -15,7 +15,7 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) if (!pte_present(orig_pte) || !pte_cont(orig_pte)) return orig_pte; - ncontig = arch_contpte_get_num_contig(ptep, + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, page_size(pte_page(orig_pte)), NULL); @@ -69,7 +69,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, int i; int ncontig; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) @@ -93,7 +93,7 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, int i, ncontig; size_t pgsize; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) __pte_clear(mm, addr, ptep); @@ -145,7 +145,80 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, int ncontig; size_t pgsize; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); return get_clear_contig(mm, addr, ptep, pgsize, ncontig); } + +/* + * huge_ptep_set_access_flags will update access flags (dirty, accesssed) + * and write permission. + * + * For a contiguous huge pte range we need to check whether or not write + * permission has to change only on the first pte in the set. Then for + * all the contiguous ptes we need to check whether or not there is a + * discrepancy between dirty or young. + */ +static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig) +{ + int i; + + if (pte_write(pte) != pte_write(__ptep_get(ptep))) + return 1; + + for (i = 0; i < ncontig; i++) { + pte_t orig_pte = __ptep_get(ptep + i); + + if (pte_dirty(pte) != pte_dirty(orig_pte)) + return 1; + + if (pte_young(pte) != pte_young(orig_pte)) + return 1; + } + + return 0; +} + +static pte_t get_clear_contig_flush(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + pte_t orig_pte = get_clear_contig(mm, addr, ptep, pgsize, ncontig); + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + + flush_tlb_range(&vma, addr, addr + (pgsize * ncontig)); + return orig_pte; +} + +int huge_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, int dirty) +{ + int ncontig; + size_t pgsize = 0; + struct mm_struct *mm = vma->vm_mm; + pte_t orig_pte; + + if (!pte_cont(pte)) + return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); + + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); + + if (!__cont_access_flags_changed(ptep, pte, ncontig)) + return 0; + + orig_pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); + + /* Make sure we don't lose the dirty or young state */ + if (pte_dirty(orig_pte)) + pte = pte_mkdirty(pte); + + if (pte_young(orig_pte)) + pte = pte_mkyoung(pte); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); + + return 1; +} From patchwork Fri Mar 21 13:06:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 14025440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8E5AC36000 for ; Fri, 21 Mar 2025 13:21:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=JOImW0XTENLZqMOVas1NKgXRQH/ppGnjlH3sp/4Pu3c=; b=BK1KjudTxdWBGt3oY4p5mS2KwQ iOOpOtnilv7/0ZU3lff3Dn48VlKkFFJKwnjoEgugZjk377TtNkwpVrzHrxaXfvqDufZG9y6J7BMzw UiOZMo41VoZgMSITyvHEKihQm7xKwhSzq7N1xjrqzSOyK7D+SXW2xnZOkKjSf0Dx3G89mZxlYdCcZ bKBf5wH9ur3SqMaq5ybLoDIfO4AWmYM3y3hzCHKORYum6iKReRtr35cxfgbwm5NhCFEdfkROgDxLw kK4Sf8Avc+iy9Ma63PhyN+6PSBl0R+avqn/YGdQWC407cCKCWZvIUQ4d8wvDUDPqffnutGqzBm6Hw h0XshOnw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvcIx-0000000EuUR-3gwD; Fri, 21 Mar 2025 13:20:55 +0000 Received: from mail-wr1-x42c.google.com ([2a00:1450:4864:20::42c]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvcD5-0000000Et5e-3N2h for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 13:14:53 +0000 Received: by mail-wr1-x42c.google.com with SMTP id ffacd0b85a97d-39130ee05b0so1860148f8f.3 for ; Fri, 21 Mar 2025 06:14:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1742562890; x=1743167690; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JOImW0XTENLZqMOVas1NKgXRQH/ppGnjlH3sp/4Pu3c=; b=14qEUqAs8amlrTOY9101Dw80jQK7ZE7yhGLYHvbTB/m9DhgZi+ItZv3Ghuv9AI+s8f qY3N861mDyCCHeIW0XbRe0VGqHqzccp7wD5YKI/Ta6I13uj0Ig3N0At3eXCIZhp0L8qT e4VJyDO/IIPgAnZzInHDOWj3iz1/J51Ko5KUaWUPsGfrx8T4rdK80AVA0ckcV59cLkvX GFpJJkCqUxR14hYztGS59jxnuKpmY4X0kIuZ4ZtkzQTKPvNI1hPhXliRil4+3+SVid76 mUFtvFSV/GLTZwWGapJmB6U12vAHCZsXLm7WP3Yss5lab/JXvcXkSL1pUgfIJ3iYM7ns gW4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742562890; x=1743167690; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JOImW0XTENLZqMOVas1NKgXRQH/ppGnjlH3sp/4Pu3c=; b=hjKW0dXrGkDV0b4CX5W1kPSwPXSSfoaLuH5uo7TS0N/JhUTEoY3x82N1OBgxAAQXc+ d1qHV2pHXTyWmeYKEJmfMd2qe0TFRUvnwy8srtvHG+bYeJhfqIvjZPEWzo1lyzUvOaGH mJj8jQ0aVMQvPVT3gdDWET35O2QAYdhUpD3cd9F0X90XhRDLkEbMsPCCuvTPjZdssAPB 06aAUC5WlTZzvdbjFTOHiZQOZ2J6a4viqf0uw+kaQcxqPCjofpdUZW4vscN6kPr/xnJR Mbq/hjgyi2gRY737aubC6IrkcPQTOA5MiHJtaWk7ZgKuLLQDNRuxEDP0ISalv/hFFZUq UVPA== X-Forwarded-Encrypted: i=1; AJvYcCWRV3tzvYSfvQa2HbO27ZVivruA3mtjd6SkI+ttNG42QbQTw7BU0lK1UIxGjkzx3K014fIxC6BtylicjyJM9pw5@lists.infradead.org X-Gm-Message-State: AOJu0Yxk5x8ZPMQCdqPFq7Q7bjyRJGY5bhJmoH8QeXW5zfzO2l6Iy8L3 CU5gxu9I1EBkahGvlCd1EZVvw4X+tZ4vTgpDMzNsbYw2xC3l+ogCsRlL9TbnxfU= X-Gm-Gg: ASbGnct2sLymC/LUPtZtkTq7/c2l9J5uxDjCHUpLVdtvYeEAg+iTigfSWrB502wQdmK 2taNbu2mOk3vbswkcqxfTsUZUUrm5mm5lMGeb9bu4l5AFgeD/PoW4fA38t/PVdeMzIAIgLeAY46 Ma07bwk2I30QC4WsPr6Nung7BwF1cPz6JEtF9r/SQjLgD7jEQEm85GthwZrB745qypDxrTbZCfR Yo2S4SV2Vdhzfqwrg8D6vRjwb+4OgOWV/wk4ZvNLAHSIJ2EgF+6pBzidvtYeUC4OY8LSXRAX9TF 3yZ55T9ycmHIDpLjzoGqVQwqcRmusBglO05AkZFJqPiJQRmxCXVT1NKcOias+scHUsTRCg== X-Google-Smtp-Source: AGHT+IG4Bh4b8ljgMGozvKhj+gGf/zTKUltGlpRU5vbjoRJl/qtI6QCkciAO8DnHWDVRjHELB4GFxw== X-Received: by 2002:a05:6000:381:b0:399:737f:4e02 with SMTP id ffacd0b85a97d-3997f938798mr3602833f8f.39.1742562890297; Fri, 21 Mar 2025 06:14:50 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3d12:52fe:c1cc:c94]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3997f9eef37sm2294241f8f.85.2025.03.21.06.14.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Mar 2025 06:14:49 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Matthew Wilcox , Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v5 8/9] mm: Use common huge_ptep_set_wrprotect() function for riscv/arm64 Date: Fri, 21 Mar 2025 14:06:34 +0100 Message-Id: <20250321130635.227011-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250321130635.227011-1-alexghiti@rivosinc.com> References: <20250321130635.227011-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250321_061451_843709_2F347695 X-CRM114-Status: GOOD ( 14.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 --- arch/arm64/mm/hugetlbpage.c | 27 --------------------------- arch/riscv/include/asm/hugetlb.h | 4 ---- arch/riscv/include/asm/pgtable.h | 7 ++++--- arch/riscv/mm/hugetlbpage.c | 22 ---------------------- include/linux/hugetlb_contpte.h | 4 ++++ mm/hugetlb_contpte.c | 20 ++++++++++++++++++++ 7 files changed, 28 insertions(+), 59 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 29a9dac52cef..f568467e8ba2 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -35,9 +35,6 @@ static inline void arch_clear_hugetlb_flags(struct folio *folio) pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT -extern void huge_ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 03cb757f7935..17f1ed34356d 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -260,33 +260,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -void huge_ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - unsigned long pfn, dpfn; - pgprot_t hugeprot; - int ncontig, i; - size_t pgsize; - pte_t pte; - - if (!pte_cont(__ptep_get(ptep))) { - __ptep_set_wrprotect(mm, addr, ptep); - return; - } - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - dpfn = pgsize >> PAGE_SHIFT; - - pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); - pte = pte_wrprotect(pte); - - hugeprot = pte_pgprot(pte); - pfn = pte_pfn(pte); - - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); -} - pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index bf533c2cef84..4c692dd82779 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -24,10 +24,6 @@ bool arch_hugetlb_migration_supported(struct hstate *h); pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); -#define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT -void huge_ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep); - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 081385e0d10a..c41b49948ee9 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -665,9 +665,8 @@ extern int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long add extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); -#define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline void __ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); } @@ -791,6 +790,8 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, #define pte_clear __pte_clear #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +#define ptep_set_wrprotect __ptep_set_wrprotect #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index b2046f4bd445..db13f7bcdd54 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -176,28 +176,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -void huge_ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep) -{ - pte_t pte = ptep_get(ptep); - pte_t orig_pte; - int pte_num; - - if (!pte_napot(pte)) { - ptep_set_wrprotect(mm, addr, ptep); - return; - } - - pte_num = arch_contpte_get_num_contig(mm, addr, ptep, 0, NULL); - - orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); - - orig_pte = pte_wrprotect(orig_pte); - - set_ptes(mm, addr, ptep, orig_pte, pte_num); -} - pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index e129578f6500..9ec8792a2f4d 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -28,4 +28,8 @@ extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty); +#define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT +extern void huge_ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index b4c409d11195..629878765081 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -222,3 +222,23 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return 1; } + +void huge_ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + int ncontig; + size_t pgsize; + pte_t pte; + + if (!pte_cont(__ptep_get(ptep))) { + __ptep_set_wrprotect(mm, addr, ptep); + return; + } + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); + pte = pte_wrprotect(pte); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); +} From patchwork Fri Mar 21 13:06:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 14025447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 945DAC36000 for ; Fri, 21 Mar 2025 13:24:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Vd9K/fWiMAnzVZgWbX0wvtztqz4Hm8uA/kr1foq/koQ=; b=qhZZ5yVHgRgKx/iW7/bW17ZL8v MvCcaOsdLi9yyCco4VhvQfb+vDaes1IlUs+5yE0on+wbyRqQFYT8lr8SdX6ZVdrbRz/0NNCm4jCY0 8RAHEHE4lzxV2U1W9ONSV3ubu9cAPDw1jDH2oDFQLfYuc1lm3Jw6Z+J6XQx6hfApH7B2hhqbAvaR/ KXz6H4QOOgQcFlM7sgotki7u08EEvkilpcyFMzJimbZn4eGyNQ/CaipnBZyBNdrIJrKvbmEQycnPd 5lRI7I7IWV/TXLaYbVGo68yKYhAA4rT4Vtlx1/bCenUiMYXcYiCs6rwZwKumoggewy8NflPKL93aV fLoHdqoQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tvcMG-0000000Ev0F-3WDq; Fri, 21 Mar 2025 13:24:20 +0000 Received: from mail-wm1-x32b.google.com ([2a00:1450:4864:20::32b]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tvcE5-0000000EtMN-1mwM for linux-arm-kernel@lists.infradead.org; Fri, 21 Mar 2025 13:15:54 +0000 Received: by mail-wm1-x32b.google.com with SMTP id 5b1f17b1804b1-43cf034d4abso20588615e9.3 for ; Fri, 21 Mar 2025 06:15:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1742562952; x=1743167752; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vd9K/fWiMAnzVZgWbX0wvtztqz4Hm8uA/kr1foq/koQ=; b=PI2LUMoW/npNaY4nzXl8yj6B9kyIkP0ropAJ//yvtWryamE5IF0+OUf7D+pX3TsxKY rQXT8zB2hcj9b/3Pb5ocbTInvPoJ0QB9g9oadiLzjLknzji3QciD/mlasEQ6G+6U5M1S 1JJ/6P19kBSVFbH5hLp+1itG73jjzRSiFNjf/TA5ZDdNGtINt7hH6igycP96cEdV8sBC XiE1njMBUVRkYep+EAxfkeSungBB1Vq+OA02vtC1N8EnlqEuUqrkzn5XENQUC3tTPK4s NbLsy22qccuOeQmqSaaHpX572G9e8l6wVKA3YDS75mNpwOZAt2V9j85aAEqiDsuEcLxG oxww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742562952; x=1743167752; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Vd9K/fWiMAnzVZgWbX0wvtztqz4Hm8uA/kr1foq/koQ=; b=LYbFPtBCfod7mFV6y2Y1w9Gf2JbxN+G29hUQPVUYi3go1kwPQvSdxAEy7zWiJOWUIf l2nudZaxgGr/z15mSS1inbfSbjBGBRTOqxOWToSwedzRI5bNDwv0NvbE9o4U38r/8+aR 0nnvilgjL6qV6GONU4V8Q1DWni2TRKF/U5GNKiuwwcMHIYmdi2RT/ikKqZqpeV4JSjgN 8DWuKU9YysHhvXsjTgUQqx8vAI3SpwDwzcZFet69imK1KuDd3aryKz3/Q/QmmAhWgFk6 di9v0dGjxlyKvgylWpbUaemToSgTFhFbuaCayN1i3LZrQmlLQS2gEkmmImh9KwGdmwiz 24Ug== X-Forwarded-Encrypted: i=1; AJvYcCXYwOImj0EcATSmEBQFVb3BwlViWyBsxnk0fM5+d2AKeEA8uXmhNCt4pPvApBouzK3KKaQK/3/ciyO1Kg2h/6dy@lists.infradead.org X-Gm-Message-State: AOJu0YyAUJDEiUUMCHmxdRYul1503tjGQsJfZCF0Syu8lxlwom36K3i8 ZnfFQMB9/zfzJaqNo8QhL7c8ccf50X4N4UJStWi1gC5e3PcfF/KpB5yQr7P4vkc= X-Gm-Gg: ASbGncsgxrXeO8y1FkebzqXFuB0CVmfJ2xL+5V8GDQ4QgjaRUM4Fy/1on6uwAasDtLb IuSftLXjyb0NhkrdDOf1D11Lnbz771+RfiWm1mZazwllxyI5MMNspLzO2cH08nx2GGe75j9bCqt KZ1m/xTE+l8/gnUpbsOvIv7oY1dNtL3WLcLev0bagtXnFDmp3vrjFfDPA1ipInhFGu+e78EiRox 9Mkm0ZzYbzjOaLAOBh5QOmr5lZJd6xnU2ePh87lJV0v0uLSXw1qgBnKvTmbRmIv4L2YCDpkNGvq 6ND20qH5e045TZmhnsM6aL8AWuokqSduUaQx2aIDvM4eB6NJSkMupszwHr/VmGe85mcjKw== X-Google-Smtp-Source: AGHT+IEeSuJJCualMyRGxNrWV663ipohlvvxYL4TpbesKMPlj8TlM2yLM3TCv4QBVXMIE7MjteYhiA== X-Received: by 2002:a05:600c:83cf:b0:43c:ec28:d310 with SMTP id 5b1f17b1804b1-43d509ec838mr36046425e9.10.1742562951783; Fri, 21 Mar 2025 06:15:51 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3d12:52fe:c1cc:c94]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-43d4fdb06b9sm26435515e9.36.2025.03.21.06.15.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Mar 2025 06:15:51 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Matthew Wilcox , Paul Walmsley , Palmer Dabbelt , Alexandre Ghiti , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v5 9/9] mm: Use common huge_ptep_clear_flush() function for riscv/arm64 Date: Fri, 21 Mar 2025 14:06:35 +0100 Message-Id: <20250321130635.227011-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250321130635.227011-1-alexghiti@rivosinc.com> References: <20250321130635.227011-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250321_061553_466700_D29910FC X-CRM114-Status: GOOD ( 16.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 -- arch/arm64/mm/hugetlbpage.c | 60 -------------------------------- arch/riscv/include/asm/hugetlb.h | 7 +--- arch/riscv/mm/hugetlbpage.c | 54 ---------------------------- include/linux/hugetlb_contpte.h | 4 +++ mm/hugetlb_contpte.c | 14 ++++++++ 6 files changed, 19 insertions(+), 123 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index f568467e8ba2..368600764127 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -35,9 +35,6 @@ static inline void arch_clear_hugetlb_flags(struct folio *folio) pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH -extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep); void __init arm64_hugetlb_cma_reserve(void); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 17f1ed34356d..08316cf4b104 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -82,52 +82,6 @@ int find_num_contig(struct mm_struct *mm, unsigned long addr, return CONT_PTES; } -/* - * Changing some bits of contiguous entries requires us to follow a - * Break-Before-Make approach, breaking the whole contiguous set - * before we can change any entries. See ARM DDI 0487A.k_iss10775, - * "Misprogramming of the Contiguous bit", page D4-1762. - * - * This helper performs the break step. - */ -static pte_t get_clear_contig(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - pte_t pte, tmp_pte; - bool present; - - pte = __ptep_get_and_clear(mm, addr, ptep); - present = pte_present(pte); - while (--ncontig) { - ptep++; - addr += pgsize; - tmp_pte = __ptep_get_and_clear(mm, addr, ptep); - if (present) { - if (pte_dirty(tmp_pte)) - pte = pte_mkdirty(pte); - if (pte_young(tmp_pte)) - pte = pte_mkyoung(pte); - } - } - return pte; -} - -static pte_t get_clear_contig_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - pte_t orig_pte = get_clear_contig(mm, addr, ptep, pgsize, ncontig); - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - - flush_tlb_range(&vma, addr, addr + (pgsize * ncontig)); - return orig_pte; -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz) { @@ -260,20 +214,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - struct mm_struct *mm = vma->vm_mm; - size_t pgsize; - int ncontig; - - if (!pte_cont(__ptep_get(ptep))) - return ptep_clear_flush(vma, addr, ptep); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); -} - static int __init hugetlbpage_init(void) { /* diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 4c692dd82779..63c7e4fa342a 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -20,14 +20,9 @@ bool arch_hugetlb_migration_supported(struct hstate *h); #endif #ifdef CONFIG_RISCV_ISA_SVNAPOT -#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH -pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep); - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte - -#endif /*CONFIG_RISCV_ISA_SVNAPOT*/ +#endif /* CONFIG_RISCV_ISA_SVNAPOT */ #include diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index db13f7bcdd54..a6176415432a 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -121,45 +121,6 @@ unsigned long hugetlb_mask_last_page(struct hstate *h) return 0UL; } -static pte_t get_clear_contig(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long ncontig) -{ - pte_t pte, tmp_pte; - bool present; - - pte = ptep_get_and_clear(mm, addr, ptep); - present = pte_present(pte); - while (--ncontig) { - ptep++; - addr += PAGE_SIZE; - tmp_pte = ptep_get_and_clear(mm, addr, ptep); - if (present) { - if (pte_dirty(tmp_pte)) - pte = pte_mkdirty(pte); - if (pte_young(tmp_pte)) - pte = pte_mkyoung(pte); - } - } - return pte; -} - -static pte_t get_clear_contig_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pte_num) -{ - pte_t orig_pte = get_clear_contig(mm, addr, ptep, pte_num); - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - bool valid = !pte_none(orig_pte); - - if (valid) - flush_tlb_range(&vma, addr, addr + (PAGE_SIZE * pte_num)); - - return orig_pte; -} - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) { unsigned long order; @@ -176,21 +137,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, - unsigned long addr, - pte_t *ptep) -{ - pte_t pte = ptep_get(ptep); - int pte_num; - - if (!pte_napot(pte)) - return ptep_clear_flush(vma, addr, ptep); - - pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); - - return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); -} - static bool is_napot_size(unsigned long size) { unsigned long order; diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index 9ec8792a2f4d..e217a3412b13 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -32,4 +32,8 @@ extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, extern void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH +extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index 629878765081..1dc211d6fbe1 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -242,3 +242,17 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, set_contptes(mm, addr, ptep, pte, ncontig, pgsize); } + +pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + struct mm_struct *mm = vma->vm_mm; + size_t pgsize; + int ncontig; + + if (!pte_cont(__ptep_get(ptep))) + return ptep_clear_flush(vma, addr, ptep); + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); +}