From patchwork Fri Mar 1 09:14:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13578190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BBADBC5475B for ; Fri, 1 Mar 2024 09:16:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SEcyODAmF6vBtuEmBmKLN3JVMVtBiBlBuA/2NdnlqXU=; b=QWe/8GLEsMwEtq oJr5pvvoZ1x6EfWTT25CsX0jq2rw0+ndZ1VYhyidltNEzaeZE1mw92Z747wA77TqhMf+GFORF2Yg6 nNkFNs8vEPhh45yKYdg9VEVQzycQ1O6jydi3n6NoNOGPIZoXakeG76uD1SMzIDvdM5g2m/3L5SZx9 Z8TlrmBoqYFTyh5UI8qvdDcHtl6rSGSqxIYMR/X8OeGpAiZ5YhvzUBMevgZ6MH2wvYCh/W1c5Ws+j PfyUHTlTiUhn/5OWR4euZiltXH9FcprGwy7p3CwG5ltzF71u/my5Jaswa3fvBPRtNeNXpgi98sXwN Hl69OQjcqwRvFZI8voTA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz0B-0000000H5dz-24kz; Fri, 01 Mar 2024 09:16:23 +0000 Received: from mail-lj1-x22e.google.com ([2a00:1450:4864:20::22e]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz05-0000000H5Z1-1LJn for linux-arm-kernel@lists.infradead.org; Fri, 01 Mar 2024 09:16:19 +0000 Received: by mail-lj1-x22e.google.com with SMTP id 38308e7fff4ca-2d33986dbc0so4495391fa.2 for ; Fri, 01 Mar 2024 01:16:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709284573; x=1709889373; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3HmYMQ4fCaWre3Z0n4nCWU3axYoW5M2HfRVImcv9VqU=; b=da/9ONvwWneT2/lkSKWululrg4BHZtzxoz+0yqj1Hwc10ZPJOgdBK96iZgLIajqXf1 3YTYtJZ5l2TmNNPPwDTVN4VTtpwJCHUOwfku+XD6MXv5beHTAn+kyz58kic8mlhvU/GJ O5cCKQQ7nfyMuX/lEJdigDpXeda3tobaVOnmpakc3Rf1johjb1lLPeEj6Es7q3Zg22PO 9D9swNALWr6ajsJKPWP6uI4zIUaqsgOlxtEdQkYtyOith9njl0CX4C+MvvqerGjlgVv/ akXxxrkqbAEivJCOs3KohKx5vKwEbrcV2RL8PAPMpnlD/pY83z5ZtBGdSv60EBEtYCLl ApTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709284573; x=1709889373; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3HmYMQ4fCaWre3Z0n4nCWU3axYoW5M2HfRVImcv9VqU=; b=n9eUL268sisfVgQMBFv0Wjw/XsuMZsNiGkfuWTLErfEW8ZZSHR9+mPDCgJ3b+G47aS aYPmi/+/u8/u/2VV1UsTkXf8MoxEISlk55Bg1+sSpx+iijutsDznZ7OUx7KA70ZRw7k/ u3h0M1temqJnMPr1OnuYJpaVQe09u8x1NM+7M5BEHAusIC1nvZBs5YThgOhpcgjp6LzH BcBmdDuyM4woM4ulKOxwD/6gR1w45Lb7bFRYh0zgrxTkHcSq9KIqq4R+T904ba6hdwK/ CVZLBdifINEIJJsX7UgJ8OkVUiMMRlKHyHltVzgTCmc5YqmcFes0T2dGsJ4kb/7XijVR Jq+A== X-Forwarded-Encrypted: i=1; AJvYcCXjODxa9WNVTCG3nMJRMdaWIwSv0Rp55LsHMlhX8O7B7KKBQgmTB/llHncbFFukUHZD+QB/tFDyAvkZuPR5FI0y6EnOZsOs8VtB8RZPZ2oPeZLQFd0= X-Gm-Message-State: AOJu0YzDIkO7Zqayrw10/3ph326F8b+5zQV4Nxlv1VU+OgvwJClEseij ECIP5fOIfxE/3xupcL3yv3nN7IzpFg4gczg3+oqqSK5K3JcJphp6KEM9VLB3lH8= X-Google-Smtp-Source: AGHT+IHQVU7ZdAsKvA/XTSgIwwIMQlWsmY+5DOvluKjOjYHWuq3/2JdBJ53cUSY5g2ynNIYVyWu0Lg== X-Received: by 2002:a05:651c:10cf:b0:2d2:eb8c:b3a6 with SMTP id l15-20020a05651c10cf00b002d2eb8cb3a6mr619531ljn.40.1709284573125; Fri, 01 Mar 2024 01:16:13 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id m21-20020a7bcb95000000b00410b0ce91b1sm7777825wmi.25.2024.03.01.01.16.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 01:16:12 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 1/9] riscv: Restore the pfn in a NAPOT pte when manipulated by core mm code Date: Fri, 1 Mar 2024 10:14:47 +0100 Message-Id: <20240301091455.246686-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240301091455.246686-1-alexghiti@rivosinc.com> References: <20240301091455.246686-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240301_011617_770990_D70FC7E7 X-CRM114-Status: GOOD ( 29.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The core mm code expects to be able to extract the pfn from a pte. NAPOT mappings work differently since its ptes actually point to the first pfn of the mapping, the other bits being used to encode the size of the mapping. So modify ptep_get() so that it returns a pte value that contains the *real* pfn (which is then different from what the HW expects) and right before storing the ptes to the page table, reset the pfn LSBs to the size of the mapping. And make sure that all NAPOT mappings are set using set_ptes(). Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/pgtable-64.h | 11 +++ arch/riscv/include/asm/pgtable.h | 105 ++++++++++++++++++++++++++-- arch/riscv/mm/hugetlbpage.c | 38 +++++----- 3 files changed, 128 insertions(+), 26 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index b42017d76924..a0f76c3071a9 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -106,6 +106,17 @@ enum napot_cont_order { #define napot_cont_mask(order) (~(napot_cont_size(order) - 1UL)) #define napot_pte_num(order) BIT(order) +static inline bool is_napot_order(unsigned int order) +{ + unsigned int napot_order; + + for_each_napot_order(napot_order) + if (order == napot_order) + return true; + + return false; +} + #ifdef CONFIG_RISCV_ISA_SVNAPOT #define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_CONT_ORDER_BASE)) #else diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 0c94260b5d0c..951f3ceb5529 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -303,6 +303,8 @@ static inline unsigned long pte_napot(pte_t pte) return pte_val(pte) & _PAGE_NAPOT; } +#define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) + static inline pte_t pte_mknapot(pte_t pte, unsigned int order) { int pos = order - 1 + _PAGE_PFN_SHIFT; @@ -312,6 +314,12 @@ static inline pte_t pte_mknapot(pte_t pte, unsigned int order) return __pte((pte_val(pte) & napot_mask) | napot_bit | _PAGE_NAPOT); } +/* pte at entry must *not* encode the mapping size in the pfn LSBs. */ +static inline pte_t pte_clear_napot(pte_t pte) +{ + return __pte(pte_val(pte) & ~_PAGE_NAPOT); +} + #else static __always_inline bool has_svnapot(void) { return false; } @@ -321,17 +329,14 @@ static inline unsigned long pte_napot(pte_t pte) return 0; } +#define pte_valid_napot(pte) false + #endif /* CONFIG_RISCV_ISA_SVNAPOT */ /* Yields the page frame number (PFN) of a page table entry */ static inline unsigned long pte_pfn(pte_t pte) { - unsigned long res = __page_val_to_pfn(pte_val(pte)); - - if (has_svnapot() && pte_napot(pte)) - res = res & (res - 1UL); - - return res; + return __page_val_to_pfn(pte_val(pte)); } #define pte_page(x) pfn_to_page(pte_pfn(x)) @@ -523,9 +528,91 @@ static inline void __set_pte_at(pte_t *ptep, pte_t pteval) set_pte(ptep, pteval); } +#ifdef CONFIG_RISCV_ISA_SVNAPOT +static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, + size_t *pgsize) +{ + pte_t __pte; + + /* We must read the raw value of the pte to get the size of the mapping */ + __pte = READ_ONCE(*ptep); + + if (pgsize) { + if (size >= PGDIR_SIZE) + *pgsize = PGDIR_SIZE; + else if (size >= P4D_SIZE) + *pgsize = P4D_SIZE; + else if (size >= PUD_SIZE) + *pgsize = PUD_SIZE; + else if (size >= PMD_SIZE) + *pgsize = PMD_SIZE; + else + *pgsize = PAGE_SIZE; + } + + /* Make sure __pte is not a swap entry */ + if (pte_valid_napot(__pte)) + return napot_pte_num(napot_cont_order(__pte)); + + return 1; +} +#endif + +static inline pte_t ptep_get(pte_t *ptep) +{ + pte_t pte = READ_ONCE(*ptep); + +#ifdef CONFIG_RISCV_ISA_SVNAPOT + /* + * The pte we load has the N bit set and the size of the mapping in + * the pfn LSBs: keep the N bit and replace the mapping size with + * the *real* pfn since the core mm code expects to find it there. + * The mapping size will be reset just before being written to the + * page table in set_ptes(). + */ + if (unlikely(pte_valid_napot(pte))) { + unsigned int order = napot_cont_order(pte); + int pos = order - 1 + _PAGE_PFN_SHIFT; + unsigned long napot_mask = ~GENMASK(pos, _PAGE_PFN_SHIFT); + pte_t *orig_ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * napot_pte_num(order)); + + pte = __pte((pte_val(pte) & napot_mask) + ((ptep - orig_ptep) << _PAGE_PFN_SHIFT)); + } +#endif + + return pte; +} +#define ptep_get ptep_get + static inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr) { +#ifdef CONFIG_RISCV_ISA_SVNAPOT + if (unlikely(pte_valid_napot(pteval))) { + unsigned int order = ilog2(nr); + + if (!is_napot_order(order)) { + /* + * Something's weird, we are given a NAPOT pte but the + * size of the mapping is not a known NAPOT mapping + * size, so clear the NAPOT bit and map this without + * NAPOT support: core mm only manipulates pte with the + * real pfn so we know the pte is valid without the N + * bit. + */ + pr_err("Incorrect NAPOT mapping, resetting.\n"); + pteval = pte_clear_napot(pteval); + } else { + /* + * NAPOT ptes that arrive here only have the N bit set + * and their pfn does not contain the mapping size, so + * set that here. + */ + pteval = pte_mknapot(pteval, order); + } + } +#endif + page_table_check_ptes_set(mm, ptep, pteval, nr); for (;;) { @@ -533,6 +620,12 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; + +#ifdef CONFIG_RISCV_ISA_SVNAPOT + if (unlikely(pte_valid_napot(pteval))) + continue; +#endif + pte_val(pteval) += 1 << _PAGE_PFN_SHIFT; } } diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 29c7606414d2..3d84fbc5c572 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -256,8 +256,7 @@ void set_huge_pte_at(struct mm_struct *mm, clear_flush(mm, addr, ptep, pgsize, pte_num); - for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) - set_pte_at(mm, addr, ptep, pte); + set_ptes(mm, addr, ptep, pte, pte_num); } int huge_ptep_set_access_flags(struct vm_area_struct *vma, @@ -267,16 +266,16 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, int dirty) { struct mm_struct *mm = vma->vm_mm; - unsigned long order; + size_t pgsize; pte_t orig_pte; - int i, pte_num; + int pte_num; if (!pte_napot(pte)) return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - order = napot_cont_order(pte); - pte_num = napot_pte_num(order); - ptep = huge_pte_offset(mm, addr, napot_cont_size(order)); + pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + ptep = huge_pte_offset(mm, addr, pte_num * pgsize); + orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); if (pte_dirty(orig_pte)) @@ -285,8 +284,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (pte_young(orig_pte)) pte = pte_mkyoung(pte); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) - set_pte_at(mm, addr, ptep, pte); + set_ptes(mm, addr, ptep, pte, pte_num); return true; } @@ -301,7 +299,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, if (!pte_napot(orig_pte)) return ptep_get_and_clear(mm, addr, ptep); - pte_num = napot_pte_num(napot_cont_order(orig_pte)); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); return get_clear_contig(mm, addr, ptep, pte_num); } @@ -311,24 +309,23 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t *ptep) { pte_t pte = ptep_get(ptep); - unsigned long order; + size_t pgsize; pte_t orig_pte; - int i, pte_num; + int pte_num; if (!pte_napot(pte)) { ptep_set_wrprotect(mm, addr, ptep); return; } - order = napot_cont_order(pte); - pte_num = napot_pte_num(order); - ptep = huge_pte_offset(mm, addr, napot_cont_size(order)); + pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + ptep = huge_pte_offset(mm, addr, pte_num * pgsize); + orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) - set_pte_at(mm, addr, ptep, orig_pte); + set_ptes(mm, addr, ptep, orig_pte, pte_num); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, @@ -341,7 +338,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_clear_flush(vma, addr, ptep); - pte_num = napot_pte_num(napot_cont_order(pte)); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } @@ -351,6 +348,7 @@ void huge_pte_clear(struct mm_struct *mm, pte_t *ptep, unsigned long sz) { + size_t pgsize; pte_t pte = ptep_get(ptep); int i, pte_num; @@ -359,8 +357,8 @@ void huge_pte_clear(struct mm_struct *mm, return; } - pte_num = napot_pte_num(napot_cont_order(pte)); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) + pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + for (i = 0; i < pte_num; i++, addr += pgsize, ptep++) pte_clear(mm, addr, ptep); } From patchwork Fri Mar 1 09:14:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13578191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79460C5478C for ; Fri, 1 Mar 2024 09:17:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WX3Q6lFI7kCR999RhGMiHKTcBCf2JFq+G/y3JKsgYkY=; b=irOajB7fZF5+9y 8E9zA7ibYOSIC/0IGBzWD3KPDhkulDSngS+DVKXRDlpJhGgcucUioHv+KlKJ+5fCwmegH7qMN3VVs f81vDiMnnuj0+2yxV/8XMc3FW3QAkyCyp2iSC/rCVUk0+KMl/5XoRzIcYNf/sCTL2RPopr+6d6NhC UG9sCem71wYMK28oiqm+w3wsegsbDVXfsz5TAEKtYqBXliJHtgrmv5WJa7jdRDaK5mpwm8T6oBi8j EQtQKQ0Pd+Jr9gNMCQJMlhzX0AcH9ogNNY8nHaK5Jh3g5gQMoo4oVMaYXZtMjffLzb+GDjFGhamek a6WQkYy1Pstpj5slkedw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz16-0000000H5uB-33xf; Fri, 01 Mar 2024 09:17:20 +0000 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz12-0000000H5s9-3a09 for linux-arm-kernel@lists.infradead.org; Fri, 01 Mar 2024 09:17:18 +0000 Received: by mail-wm1-x330.google.com with SMTP id 5b1f17b1804b1-412a3371133so13602325e9.2 for ; Fri, 01 Mar 2024 01:17:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709284634; x=1709889434; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=celeezlUFfnCgdabriQL+vY9x87nEUlib9jbsT9nzg0=; b=3bJaS0ryaL9sbB9ktQti0IYObfKb0u5BFSdgqq4ilx+WQDz7DslhrfnFAkRnCwR2Yg PRWOqE4+WccSUBvVj3+metU2BlVcQu1QtvI9VTcq/JDyvaENZkiYBW6Cy2CWJyB+red0 ygHXlG3KCQCuFMau++ecDfJewDpnbcaqsFSuBzUT7bQ1DW1BK4/5ZsJLLiMq7n3oFGZU X6hnY3eJHYlPZStQ1OE8khZIAJM2EO8ApDV6pdVs2L5uGk3ACwHUcEywhKpu4i0ebJfb +7g0hBIXg3O2Ay6J3YUf7FrVk3fpznZ0Qoqnhh6o9Cn+eo+eV5HiCPzzajkEvzkzf4fU qYKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709284634; x=1709889434; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=celeezlUFfnCgdabriQL+vY9x87nEUlib9jbsT9nzg0=; b=pSiNbO+yP+/l7fx/D86I/AN9TK9bdGJVub8mhasNCbKqIKBOQ4LUiAaT/nburW86wJ NrdcihDVxWgo64RcBTJsEebC2kT1wBKrBVQ5XonCeJcFmYBM/sOdHM+OBQoMTMLBz6hc T9RHyafTIY/7hOdnUAFBpul9vzUQ8wircNfQoeoWIPsjdS2meVUEpPT+NFGBDBPciT2Q 6GP0JYhYtb+uJv3LUQSHz2LyTnG4s8q2brwFAS7ixPgDyps7qEmTm+6i2WL3GjQYIF1l wHsvcmKR9o9XUWLgP8OccV8Vr+i8HVe1k72l+SWQR7bSvw7y+2gqYsPf6Jtyc8kvqq2U 4hbA== X-Forwarded-Encrypted: i=1; AJvYcCUYPw5i/ou+tvNwds9oewA9bj7zG9Sq2KiKkN6dzdaoG8jcnFslCn1ZBxovop7nVWxuaE2eBGEd22e7MbVtpzj6oAC5/E6Kzcn56wT7g7tdgRDL22U= X-Gm-Message-State: AOJu0Ywy/uz7fUf9OEwLOonE3+UEvnJGFZpqQGaaGMrgWwDY1jZnubyu 1MBFWLv4dDqCvU0Q0wRIvzj+MJX0pnTkr8/VX8g0TA59mC13MckFKQ5rkbQqn+k= X-Google-Smtp-Source: AGHT+IFB0pFFxU/+IPg3Lb+DpLxmFSKXoGv4IrPSAZpcxiUzICaAY8iLg+9vVvkiiGxUNIP5Ue4QKg== X-Received: by 2002:a05:600c:3b26:b0:412:c810:fd18 with SMTP id m38-20020a05600c3b2600b00412c810fd18mr995837wms.18.1709284634158; Fri, 01 Mar 2024 01:17:14 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id l33-20020a05600c1d2100b00412ca88537dsm358022wms.0.2024.03.01.01.17.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 01:17:13 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 2/9] riscv: Safely remove huge_pte_offset() when manipulating NAPOT ptes Date: Fri, 1 Mar 2024 10:14:48 +0100 Message-Id: <20240301091455.246686-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240301091455.246686-1-alexghiti@rivosinc.com> References: <20240301091455.246686-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240301_011716_939892_E58D6CCF X-CRM114-Status: GOOD ( 10.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The pte_t pointer is expected to point to the first entry of the NAPOT mapping so no need to use huge_pte_offset(), similarly to what is done in arm64. Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/hugetlbpage.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 3d84fbc5c572..2477d20c1497 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -274,7 +274,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return ptep_set_access_flags(vma, addr, ptep, pte, dirty); pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); - ptep = huge_pte_offset(mm, addr, pte_num * pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); @@ -319,10 +318,8 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, } pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); - ptep = huge_pte_offset(mm, addr, pte_num * pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); - orig_pte = pte_wrprotect(orig_pte); set_ptes(mm, addr, ptep, orig_pte, pte_num); From patchwork Fri Mar 1 09:14:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13578192 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B540CC5475B for ; Fri, 1 Mar 2024 09:18:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5axPEmYv9/vAepg9RjcP/25rko81EscocYUs0MMqobk=; b=nzl+/au60+gBax tSRE0mTNb5f5C3dpzo9ZU1dlnIALHCHMOquOz4k41pJmNXwdoTLnhvx8Q3OGtm7JL8ffSuhYrQlx9 2E2sQnI2zoMuqASHxnR0NRmQFDMW0semw9U86HsThruIeOkje4kAizpqevGCuJFUH2nqRsq3Wd6Yi yFqZTpmdsEe1ErHsMq/cLh1Mft2iSSQn8mYLhAvFcDSRZs7IG8W9I/MMhJ/r4sDrC8Cm/HVTuhW0q LnigERI/4etiSWDtgEoywizowRPW2rDUj3zwUwnAH+7fxmIjT4YLeSXWo9NOxzXjL2omambatkguT ho7iCFytOptT0dpZZbSA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz2A-0000000H6B8-3Ngk; Fri, 01 Mar 2024 09:18:26 +0000 Received: from mail-lj1-x22d.google.com ([2a00:1450:4864:20::22d]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz22-0000000H693-0fxJ for linux-arm-kernel@lists.infradead.org; Fri, 01 Mar 2024 09:18:23 +0000 Received: by mail-lj1-x22d.google.com with SMTP id 38308e7fff4ca-2d24a727f78so21197501fa.0 for ; Fri, 01 Mar 2024 01:18:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709284695; x=1709889495; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f1tNyFTcsXe5j/Xz9tdKUfPkAneqDazbYHB6YPwtZWk=; b=Ch7rfv/pV3JaoCdz723BvASm7DaG2IDTB/XUCoIpB6GrySLSEITZdaVAyzQbjqjdSv nmNf8zjYZkJ3vSvXJrmfk0t9C2z47I+ZkJhj1gB8gEwzD1qmr+IhN9WtmOc85SuGwz6P vUTix7QyZ0YEm97JRtus+Bx0XPTVswLXXRuYhVvxaqF7bMpc66KlruprNqmWBFuq/F5j h6gXAU6m83vfqyIwIveZdZmv89QZRmdTUskigUoJQPUO486dIEnuKybA4vAsDj7gAD3Y zqA/29fWWjpEgRWbPhE4ZbtxstxgINGZVA2sr4DBbfFOMTUORCExuXnccQ4ezqa9PR2Y X1/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709284695; x=1709889495; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f1tNyFTcsXe5j/Xz9tdKUfPkAneqDazbYHB6YPwtZWk=; b=tz2j56qgpwIz+omEYalQwE8RyxHKPQeKthrKzZmz9MD77Vzcw5ByJP/Tp7nd91M1Vd eX43v9y+mmUPT4VZH45A8RV7KlQLn490OIThrtgK/Ga9Dwwta54aXEmJTUIVDoSGz6Et RhY1VwhyEZqYfzpNy410VTYmfucyd17Z1IJOe1P1U23kQ4PrbkKdFDzrisCHYYS6gxxe nyTK8hkdYiG2p6wxG4dLULXCUnTtuqxe1m6VEjsclMJPtCjDuyw3djll27qitG4l18Q2 bquw3ANx2mUsIw5397MO+zfggII3H9yBEqnTGyoYboKOUNUJDGiVfc0sWt8Q4SQbEAL7 a29A== X-Forwarded-Encrypted: i=1; AJvYcCVJf1EvX0/AQexN24XmHNcci7t+aH/L+Rd46x7kM5lJKj89OMDOze5Cz4IpyeqitR9EVXwZ5SiwWc3hLSv1CTm4GezuNxdrDdBxf3amBoRT0J1itkg= X-Gm-Message-State: AOJu0YzpzFNDmw1rpQoB05ZSzzWDL0MNNbIYH64vy4ufXjP26S+MTJDj qJsYabrJo+6ruTvbZ1irEDpGAeajs2neg79lH0kOU4R06u8+g1Eld6qIw603Nzo= X-Google-Smtp-Source: AGHT+IGpqFyv1mrLhJ3nW6qi9rLC4OBLYLmhrJ+wNsqKGTezYuRrvThKSI3wD+ZkJxl8NSkgNWNgvg== X-Received: by 2002:a05:651c:b1e:b0:2d2:cb34:2e0c with SMTP id b30-20020a05651c0b1e00b002d2cb342e0cmr1088079ljr.10.1709284695243; Fri, 01 Mar 2024 01:18:15 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id m20-20020a7bca54000000b0041290251dc2sm7744159wml.14.2024.03.01.01.18.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 01:18:14 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 3/9] mm: Use common huge_ptep_get() function for riscv/arm64 Date: Fri, 1 Mar 2024 10:14:49 +0100 Message-Id: <20240301091455.246686-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240301091455.246686-1-alexghiti@rivosinc.com> References: <20240301091455.246686-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240301_011818_436620_3D2AD852 X-CRM114-Status: GOOD ( 23.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For that, we need to introduce: - a new config: ARCH_HAS_CONTPTE, - a new arch specific function which returns the number of contiguous PTE in a mapping and its base page size, - a pte_cont() helper, only introduced for riscv since we keep the arm64 naming (contpte) which is more explicit than the riscv's (napot). Signed-off-by: Alexandre Ghiti --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/pgtable.h | 31 ++++++++++++++++++ arch/arm64/mm/hugetlbpage.c | 55 ++------------------------------ arch/riscv/Kconfig | 1 + arch/riscv/include/asm/hugetlb.h | 2 +- arch/riscv/include/asm/pgtable.h | 1 + arch/riscv/mm/hugetlbpage.c | 24 -------------- mm/Kconfig | 3 ++ mm/Makefile | 1 + mm/contpte.c | 45 ++++++++++++++++++++++++++ 10 files changed, 86 insertions(+), 78 deletions(-) create mode 100644 mm/contpte.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index aa7c1d435139..5e6bd49169d7 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -20,6 +20,7 @@ config ARM64 select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_CACHE_LINE_SIZE + select ARCH_HAS_CONTPTE select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL select ARCH_HAS_DEBUG_VM_PGTABLE diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 79ce70fbb751..3003a10547de 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1124,6 +1124,37 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma, extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); + +static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, + size_t *pgsize) +{ + int contig_ptes = 0; + + *pgsize = size; + + switch (size) { +#ifndef __PAGETABLE_PMD_FOLDED + case PUD_SIZE: + if (pud_sect_supported()) + contig_ptes = 1; + break; +#endif + case PMD_SIZE: + contig_ptes = 1; + break; + case CONT_PMD_SIZE: + *pgsize = PMD_SIZE; + contig_ptes = CONT_PMDS; + break; + case CONT_PTE_SIZE: + *pgsize = PAGE_SIZE; + contig_ptes = CONT_PTES; + break; + } + + return contig_ptes; +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 8116ac599f80..6b61714d7726 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -119,57 +119,6 @@ static int find_num_contig(struct mm_struct *mm, unsigned long addr, return CONT_PTES; } -static inline int num_contig_ptes(unsigned long size, size_t *pgsize) -{ - int contig_ptes = 0; - - *pgsize = size; - - switch (size) { -#ifndef __PAGETABLE_PMD_FOLDED - case PUD_SIZE: - if (pud_sect_supported()) - contig_ptes = 1; - break; -#endif - case PMD_SIZE: - contig_ptes = 1; - break; - case CONT_PMD_SIZE: - *pgsize = PMD_SIZE; - contig_ptes = CONT_PMDS; - break; - case CONT_PTE_SIZE: - *pgsize = PAGE_SIZE; - contig_ptes = CONT_PTES; - break; - } - - return contig_ptes; -} - -pte_t huge_ptep_get(pte_t *ptep) -{ - int ncontig, i; - size_t pgsize; - pte_t orig_pte = ptep_get(ptep); - - if (!pte_present(orig_pte) || !pte_cont(orig_pte)) - return orig_pte; - - ncontig = num_contig_ptes(page_size(pte_page(orig_pte)), &pgsize); - for (i = 0; i < ncontig; i++, ptep++) { - pte_t pte = ptep_get(ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - return orig_pte; -} - /* * Changing some bits of contiguous entries requires us to follow a * Break-Before-Make approach, breaking the whole contiguous set @@ -250,7 +199,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, unsigned long pfn, dpfn; pgprot_t hugeprot; - ncontig = num_contig_ptes(sz, &pgsize); + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) @@ -397,7 +346,7 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, int i, ncontig; size_t pgsize; - ncontig = num_contig_ptes(sz, &pgsize); + ncontig = arch_contpte_get_num_contig(NULL, sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) pte_clear(mm, addr, ptep); diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index dba28a756e63..121183768d1a 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -19,6 +19,7 @@ config RISCV select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2 select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE select ARCH_HAS_BINFMT_FLAT + select ARCH_HAS_CONTPTE if RISCV_ISA_SVNAPOT select ARCH_HAS_CURRENT_STACK_POINTER select ARCH_HAS_DEBUG_VIRTUAL if MMU select ARCH_HAS_DEBUG_VM_PGTABLE diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 20f9c3ba2341..a431a0c0e0fa 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -47,7 +47,7 @@ pte_t huge_ptep_get(pte_t *ptep); pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#endif /*CONFIG_RISCV_ISA_SVNAPOT*/ +#endif /* CONFIG_RISCV_ISA_SVNAPOT */ #include diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 951f3ceb5529..ddff4a56e12d 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -302,6 +302,7 @@ static inline unsigned long pte_napot(pte_t pte) { return pte_val(pte) & _PAGE_NAPOT; } +#define pte_cont pte_napot #define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 2477d20c1497..51ec80cf2028 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -3,30 +3,6 @@ #include #ifdef CONFIG_RISCV_ISA_SVNAPOT -pte_t huge_ptep_get(pte_t *ptep) -{ - unsigned long pte_num; - int i; - pte_t orig_pte = ptep_get(ptep); - - if (!pte_present(orig_pte) || !pte_napot(orig_pte)) - return orig_pte; - - pte_num = napot_pte_num(napot_cont_order(orig_pte)); - - for (i = 0; i < pte_num; i++, ptep++) { - pte_t pte = ptep_get(ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - - return orig_pte; -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, diff --git a/mm/Kconfig b/mm/Kconfig index ffc3a2ba3a8c..71d92e6c50d9 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1001,6 +1001,9 @@ config IDLE_PAGE_TRACKING config ARCH_HAS_CACHE_LINE_SIZE bool +config ARCH_HAS_CONTPTE + bool + config ARCH_HAS_CURRENT_STACK_POINTER bool help diff --git a/mm/Makefile b/mm/Makefile index e4b5b75aaec9..d5aa9326fc80 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -90,6 +90,7 @@ obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o +obj-$(CONFIG_ARCH_HAS_CONTPTE) += contpte.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o ifdef CONFIG_SWAP diff --git a/mm/contpte.c b/mm/contpte.c new file mode 100644 index 000000000000..c3f4b8039b19 --- /dev/null +++ b/mm/contpte.c @@ -0,0 +1,45 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 Rivos Inc. + */ + +#include +#include +#include + +/* + * Any arch that wants to use that needs to define: + * * contpte macros + * - pte_cont() + * - arch_contpte_get_num_contig() + */ + +/* + * This file implements the following contpte aware API: + * huge_ptep_get() + */ + +pte_t huge_ptep_get(pte_t *ptep) +{ + int ncontig, i; + size_t pgsize; + pte_t orig_pte = ptep_get(ptep); + + if (!pte_present(orig_pte) || !pte_cont(orig_pte)) + return orig_pte; + + ncontig = arch_contpte_get_num_contig(ptep, + page_size(pte_page(orig_pte)), + &pgsize); + + for (i = 0; i < ncontig; i++, ptep++) { + pte_t pte = ptep_get(ptep); + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + return orig_pte; +} From patchwork Fri Mar 1 09:14:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13578205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BE534C5475B for ; Fri, 1 Mar 2024 09:19:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=cv3jiuAnEmEEFwkMMIS4kVv1YT5nNUMeSk3WkuibXAQ=; b=fnPqVvHqwKH9jf wF/OrrsgVPsUCgwu3e/uEGxt4sugG0GSmxz1iKNJvu6xhvoJ+yZra1mcNNKsZDCDunBhpYmLQH2M1 hLu6vxTMb/41fC+8xfRClyb9hWcRSGIXHrzUFT3Oknb06QTgutG6k13bjqhtpnzpEpID/F0jrbFeG OsOHhhs05y2okbizDnz5/KGXCIwhxJg8qAV/S7C4d0VFRuILveJ687Ar4IvvGzoxB/iZC7SeI60gz bS7ZfSB7zqiIENUs7imvmjY4D7cbWSua0i7f9rWKcAsPuxggs/HEC+QPEKxKNEW0a9wxDuyUMUR4f AJ+DGNbdkx6ngmxlKfIA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz32-0000000H6RV-3qIv; Fri, 01 Mar 2024 09:19:20 +0000 Received: from mail-wm1-x336.google.com ([2a00:1450:4864:20::336]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz30-0000000H6Pa-07iM for linux-arm-kernel@lists.infradead.org; Fri, 01 Mar 2024 09:19:19 +0000 Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-412bb23e5c5so10024415e9.1 for ; Fri, 01 Mar 2024 01:19:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709284756; x=1709889556; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mB8cpSFUHI2v28tFlR/YUHSb5lk45+n/9IYk3vd50wk=; b=yO44BNRqjdmhweskNbJYRv+vg2WQJ6F+t1LZ4Em/vJKWChXq6X4JeN1W9AXqs4G+qj 953cIWfP57c2WoUXa67fDM10tg6EtUDKE0G9Y3y3O3oO+ZC5RBuQexWgBUraOqRuADVK tpCgqDE7NxipQij8O+tUW5dIoWPh8N5S/Ft8PLZ64N71kiVP0SvuR5kAJZNNEkkZMeKz S4fENVJPhu2Zl7Pc98U1jET7o5zPUg509mJS/dQoR1PVgH0U40OTJx779hyRJpesqA6M gxZCkvPoEmZmuWsjThdGyJppVSWUxfyGO4Qnr8FHBVePGPnjez3e8U2FgF/PmzyUnDgv PfOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709284756; x=1709889556; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mB8cpSFUHI2v28tFlR/YUHSb5lk45+n/9IYk3vd50wk=; b=kQq7tCL/GLLIrDbK6xac50IBt8dulpmiebMgL4iFtd7ix9YFHsOyKGpUvnWdhbSYkx gnRIXDfICuEbReCLjgv31zaCbwcjudJ8XZz2GY/C8kPerbjjCPvKKYT02q1JCAqwtneo feZgNZoir66ln8e4s9k2V3WTh06sguO8a0nmCPaUTHRkORIFjnCT9Fxe2VMui67rga2h KWy2IeTlIqHXCAhk5pBHn6LGzzHIRl4FMAAdRmXtIz/fqWKnQCA8gVhwz66pEZJ6zkdo ImyfFN5xHOwEYS9Y+MVvRggxY6whI4+Y9785OkXnztKnFXJVJp9XHeaQzghM3YVXzSj4 Z4qg== X-Forwarded-Encrypted: i=1; AJvYcCWFNAlKQFWmkkJigZ0ETb3mrOk8L4HjIg39kFeFNp64W0So7/1Te1Z/socvEJoN5v0N7hN7D9IE9cz2ZRsqYBJtmnaAZnttXfYvfBLb+F7ryfyll2Q= X-Gm-Message-State: AOJu0YyKn29sDPGXOCJoBF5HdD8GgSURUitnTOH6PPhBUrEyVk8fsCNs LIJ0/GAjbJEWCvUVM1kFe77tiimU2pbDYVxFJW9Rv7Znq5HSWoDog67gpZtbPF8clXHwabjEQTG 4 X-Google-Smtp-Source: AGHT+IHpEyM7pTJk28YugLtMmnzHnlZqFOrfhIPrp2GM/yv98BwZmlX0jtBa8/Iqo86IkfuCuT/7uw== X-Received: by 2002:adf:9dca:0:b0:33e:1a3f:44b4 with SMTP id q10-20020adf9dca000000b0033e1a3f44b4mr1239957wre.26.1709284756335; Fri, 01 Mar 2024 01:19:16 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id bv16-20020a0560001f1000b0033e0523b829sm4144440wrb.13.2024.03.01.01.19.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 01:19:16 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 4/9] mm: Use common set_huge_pte_at() function for riscv/arm64 Date: Fri, 1 Mar 2024 10:14:50 +0100 Message-Id: <20240301091455.246686-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240301091455.246686-1-alexghiti@rivosinc.com> References: <20240301091455.246686-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240301_011918_109475_D3A83CBC X-CRM114-Status: GOOD ( 23.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 16 ++++++--- arch/arm64/mm/hugetlbpage.c | 56 ----------------------------- arch/riscv/include/asm/pgtable.h | 26 ++++++++++---- arch/riscv/mm/hugetlbpage.c | 62 -------------------------------- mm/contpte.c | 58 ++++++++++++++++++++++++++++++ 5 files changed, 88 insertions(+), 130 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 3003a10547de..437e9638b2b9 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -341,9 +341,10 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) mte_sync_tags(pte, nr_pages); } -static inline void set_ptes(struct mm_struct *mm, - unsigned long __always_unused addr, - pte_t *ptep, pte_t pte, unsigned int nr) +static inline void __set_ptes(struct mm_struct *mm, + unsigned long __always_unused addr, + pte_t *ptep, pte_t pte, unsigned int nr, + unsigned long pgsize) { page_table_check_ptes_set(mm, ptep, pte, nr); __sync_cache_and_tags(pte, nr); @@ -354,10 +355,15 @@ static inline void set_ptes(struct mm_struct *mm, if (--nr == 0) break; ptep++; - pte_val(pte) += PAGE_SIZE; + pte_val(pte) += pgsize; } } -#define set_ptes set_ptes + +#define set_ptes(mm, addr, ptep, pte, nr) \ + __set_ptes(mm, addr, ptep, pte, nr, PAGE_SIZE) + +#define set_contptes(mm, addr, ptep, pte, nr, pgsize) \ + __set_ptes(mm, addr, ptep, pte, nr, pgsize) /* * Huge pte definitions. diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 6b61714d7726..4da951e81bde 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -166,62 +166,6 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm, return orig_pte; } -/* - * Changing some bits of contiguous entries requires us to follow a - * Break-Before-Make approach, breaking the whole contiguous set - * before we can change any entries. See ARM DDI 0487A.k_iss10775, - * "Misprogramming of the Contiguous bit", page D4-1762. - * - * This helper performs the break step for use cases where the - * original pte is not needed. - */ -static void clear_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long i, saddr = addr; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - ptep_clear(mm, addr, ptep); - - flush_tlb_range(&vma, saddr, addr); -} - -void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned long sz) -{ - size_t pgsize; - int i; - int ncontig; - unsigned long pfn, dpfn; - pgprot_t hugeprot; - - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); - - if (!pte_present(pte)) { - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - set_pte_at(mm, addr, ptep, pte); - return; - } - - if (!pte_cont(pte)) { - set_pte_at(mm, addr, ptep, pte); - return; - } - - pfn = pte_pfn(pte); - dpfn = pgsize >> PAGE_SHIFT; - hugeprot = pte_pgprot(pte); - - clear_flush(mm, addr, ptep, pgsize, ncontig); - - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index ddff4a56e12d..03f8ced8b26a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -533,29 +533,39 @@ static inline void __set_pte_at(pte_t *ptep, pte_t pteval) static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, size_t *pgsize) { + unsigned long hugepage_shift; pte_t __pte; /* We must read the raw value of the pte to get the size of the mapping */ __pte = READ_ONCE(*ptep); - if (pgsize) { - if (size >= PGDIR_SIZE) + if (size >= PGDIR_SIZE) { + if (pgsize) *pgsize = PGDIR_SIZE; - else if (size >= P4D_SIZE) + hugepage_shift = PGDIR_SHIFT; + } else if (size >= P4D_SIZE) { + if (pgsize) *pgsize = P4D_SIZE; - else if (size >= PUD_SIZE) + hugepage_shift = P4D_SHIFT; + } else if (size >= PUD_SIZE) { + if (pgsize) *pgsize = PUD_SIZE; - else if (size >= PMD_SIZE) + hugepage_shift = PUD_SHIFT; + } else if (size >= PMD_SIZE) { + if (pgsize) *pgsize = PMD_SIZE; - else + hugepage_shift = PMD_SHIFT; + } else { + if (pgsize) *pgsize = PAGE_SIZE; + hugepage_shift = PAGE_SHIFT; } /* Make sure __pte is not a swap entry */ if (pte_valid_napot(__pte)) return napot_pte_num(napot_cont_order(__pte)); - return 1; + return size >> hugepage_shift; } #endif @@ -631,6 +641,8 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, } } #define set_ptes set_ptes +#define set_contptes(mm, addr, ptep, pte, nr, pgsize) \ + set_ptes(mm, addr, ptep, pte, nr) static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 51ec80cf2028..ebc735f5d325 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -173,68 +173,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -static void clear_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long i, saddr = addr; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - ptep_get_and_clear(mm, addr, ptep); - - flush_tlb_range(&vma, saddr, addr); -} - -/* - * When dealing with NAPOT mappings, the privileged specification indicates that - * "if an update needs to be made, the OS generally should first mark all of the - * PTEs invalid, then issue SFENCE.VMA instruction(s) covering all 4 KiB regions - * within the range, [...] then update the PTE(s), as described in Section - * 4.2.1.". That's the equivalent of the Break-Before-Make approach used by - * arm64. - */ -void set_huge_pte_at(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - pte_t pte, - unsigned long sz) -{ - unsigned long hugepage_shift, pgsize; - int i, pte_num; - - if (sz >= PGDIR_SIZE) - hugepage_shift = PGDIR_SHIFT; - else if (sz >= P4D_SIZE) - hugepage_shift = P4D_SHIFT; - else if (sz >= PUD_SIZE) - hugepage_shift = PUD_SHIFT; - else if (sz >= PMD_SIZE) - hugepage_shift = PMD_SHIFT; - else - hugepage_shift = PAGE_SHIFT; - - pte_num = sz >> hugepage_shift; - pgsize = 1 << hugepage_shift; - - if (!pte_present(pte)) { - for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) - set_ptes(mm, addr, ptep, pte, 1); - return; - } - - if (!pte_napot(pte)) { - set_ptes(mm, addr, ptep, pte, 1); - return; - } - - clear_flush(mm, addr, ptep, pgsize, pte_num); - - set_ptes(mm, addr, ptep, pte, pte_num); -} - int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, diff --git a/mm/contpte.c b/mm/contpte.c index c3f4b8039b19..f7bfa861c6a1 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -12,11 +12,13 @@ * * contpte macros * - pte_cont() * - arch_contpte_get_num_contig() + * - set_contptes() */ /* * This file implements the following contpte aware API: * huge_ptep_get() + * set_huge_pte_at() */ pte_t huge_ptep_get(pte_t *ptep) @@ -43,3 +45,59 @@ pte_t huge_ptep_get(pte_t *ptep) } return orig_pte; } + +/* + * ARM64: Changing some bits of contiguous entries requires us to follow a + * Break-Before-Make approach, breaking the whole contiguous set + * before we can change any entries. See ARM DDI 0487A.k_iss10775, + * "Misprogramming of the Contiguous bit", page D4-1762. + * + * RISCV: When dealing with NAPOT mappings, the privileged specification + * indicates that "if an update needs to be made, the OS generally should first + * mark all of the PTEs invalid, then issue SFENCE.VMA instruction(s) covering + * all 4 KiB regions within the range, [...] then update the PTE(s), as + * described in Section 4.2.1.". That's the equivalent of the Break-Before-Make + * approach used by arm64. + * + * This helper performs the break step for use cases where the + * original pte is not needed. + */ +static void clear_flush(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long i, saddr = addr; + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) + ptep_clear(mm, addr, ptep); + + flush_tlb_range(&vma, saddr, addr); +} + +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned long sz) +{ + size_t pgsize; + int i; + int ncontig; + + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + + if (!pte_present(pte)) { + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + set_pte_at(mm, addr, ptep, pte); + return; + } + + if (!pte_cont(pte)) { + set_pte_at(mm, addr, ptep, pte); + return; + } + + clear_flush(mm, addr, ptep, pgsize, ncontig); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); +} From patchwork Fri Mar 1 09:14:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13578206 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9F730C5478C for ; Fri, 1 Mar 2024 09:20:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=abJAnMVAps+/0JIsa+YuR2M9wa8pZ3ijIyQVw7SOULs=; b=a5lmcEWSTJsvYU t4LVQZ4iqOvwKziKxdK4I2l3Q3vO6lgi+05m8/aucYon9uDC9mDH38kLhlZ7w2Z+QgXUVMuQEWNoz El410rABiYd0YU6gkRApXN8R0U4Qkvr5+/VYO9xbv4mK092uJoaxqIZvkIDtDVCtUYQuUsIpvVnHF FoybSNHT4l5N6k2ooZSgKkrZl7e30WwTTdeY0Q3+upLnYdP8lipPKWqmravcbW5/uR45bqC3ecmTz dPWDOxsP/cd1e0WAMMTyVtHhCSMjGoh/22IdjMsQuWjqNMkY6us474joFNLLbbTQFn2zeWgmZ9/2X WDmFaY+zZNTa/5qFx25g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz49-0000000H77L-1M2J; Fri, 01 Mar 2024 09:20:29 +0000 Received: from mail-wm1-x32a.google.com ([2a00:1450:4864:20::32a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz40-0000000H6xT-3GLc for linux-arm-kernel@lists.infradead.org; Fri, 01 Mar 2024 09:20:27 +0000 Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-412a9e9c776so17909515e9.0 for ; Fri, 01 Mar 2024 01:20:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709284817; x=1709889617; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rdWlfuEJdB5bVZdrcri04Zmq/yhRfMcRDdftEw7qNFA=; b=zHgGYirN0xvMYJmkxwArVxKkMXXRfxx74Pgp5W9NM3GUNkTkyt5DI4h6AZ3USv+ZGa EbyzEsCH5/kjWHqtuwKQpYLbCWRZ+xotegyhHhBHviihaXTahk3mf4zkdep3St71d4KU f9rUKU56qmIpaKBFmsdAJQzAUeqVAIE3d2oXTNV2jIQTt0jVGuTenEfEj5PHmthPDQBW I4Cz81GjdTWm9pVs9XNrTg5c2mAesln0OJauGN1OA3IRjNPLThjDRlXqR6o0yyJOodjc V5wHk9GfTP+KfqjJkMKvOr7FpDupZE2SRi8gXuncUf3QHv6zmuwqtAXQCm7VQUKY8dHT o4Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709284817; x=1709889617; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rdWlfuEJdB5bVZdrcri04Zmq/yhRfMcRDdftEw7qNFA=; b=JSHhKrxXrXH2vuK3nwj1eqryZGnDYiBGRjNO55/zmEY+GFeCXu5Hiv0wPHjWdnwWn7 ZjEV3xbR1pfReu2r1+O4phXJ+x/b7WYNxJeN3nprfjq3ce8ErvyzVlPOOAoNFBepVvK+ y7aMds0EdR1J7n9OvoHj7CvYGlq0nosqF1AG6/1vx/CUjj5XwHELUUqb5wuH9aiFMAZG dCYtzikuiGarSUCcA6OipT1bigcB7K62qSqaBZzmtyqbyw6NmASVKJYydCJn+rxDMTgG DL9EnznNOWgOFi42Tnc4pTBuBaTRhfvzwm44Ry1JRHGojlhEYfs7yyVtdItTslHImWKU JPqg== X-Forwarded-Encrypted: i=1; AJvYcCWzRC0JaGHrZu28eNHFKg1YCDVz4GNTa9hWkneYlCoA3FLlEP8GgbZNphP+JwpQiPDrXQmZVAY1LP402N+742mf0S/qHzF38VP66Q1xyYpOROvbYqM= X-Gm-Message-State: AOJu0YzUfvPvA7rvEFjLnmtMcBW2eDjuO4C94p+ohWEp2/uTvaXL8xJ7 0MnbEvwBAHlG+8c81UzKQYyCYCjcaFTCv7ejzfsS9lXObciPs9zTVNA37ItSNx8= X-Google-Smtp-Source: AGHT+IHirYUWWAQ3EzQ3D3sM4w5kbNmWdiMfyHww76VDivA5fk7+UlmhKkRo6vORNs4FOsSe8ylbjw== X-Received: by 2002:adf:a3c9:0:b0:33e:21bc:bc6f with SMTP id m9-20020adfa3c9000000b0033e21bcbc6fmr809159wrb.11.1709284817327; Fri, 01 Mar 2024 01:20:17 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id l9-20020a056000022900b0033cf2063052sm3994161wrz.111.2024.03.01.01.20.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 01:20:17 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 5/9] mm: Use common huge_pte_clear() function for riscv/arm64 Date: Fri, 1 Mar 2024 10:14:51 +0100 Message-Id: <20240301091455.246686-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240301091455.246686-1-alexghiti@rivosinc.com> References: <20240301091455.246686-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240301_012021_054871_33A49D2B X-CRM114-Status: GOOD ( 13.58 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Both architectures have the same implementation so move it to generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/mm/hugetlbpage.c | 12 ------------ arch/riscv/mm/hugetlbpage.c | 19 ------------------- mm/contpte.c | 13 +++++++++++++ 3 files changed, 13 insertions(+), 31 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 4da951e81bde..09b55bac8c7c 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -284,18 +284,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -void huge_pte_clear(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long sz) -{ - int i, ncontig; - size_t pgsize; - - ncontig = arch_contpte_get_num_contig(NULL, sz, &pgsize); - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - pte_clear(mm, addr, ptep); -} - pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index ebc735f5d325..01b1403dd4cb 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -254,25 +254,6 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } -void huge_pte_clear(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long sz) -{ - size_t pgsize; - pte_t pte = ptep_get(ptep); - int i, pte_num; - - if (!pte_napot(pte)) { - pte_clear(mm, addr, ptep); - return; - } - - pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); - for (i = 0; i < pte_num; i++, addr += pgsize, ptep++) - pte_clear(mm, addr, ptep); -} - static bool is_napot_size(unsigned long size) { unsigned long order; diff --git a/mm/contpte.c b/mm/contpte.c index f7bfa861c6a1..5200c234404d 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -19,6 +19,7 @@ * This file implements the following contpte aware API: * huge_ptep_get() * set_huge_pte_at() + * huge_pte_clear() */ pte_t huge_ptep_get(pte_t *ptep) @@ -101,3 +102,15 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, set_contptes(mm, addr, ptep, pte, ncontig, pgsize); } + +void huge_pte_clear(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned long sz) +{ + int i, ncontig; + size_t pgsize; + + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) + pte_clear(mm, addr, ptep); +} From patchwork Fri Mar 1 09:14:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13578207 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9691AC5475B for ; Fri, 1 Mar 2024 09:21:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=BohJKPJMjYBcEHSqdRxgE8vbrVzqGfujXmqEtdE53nc=; b=uo2J41k7GMRIme h08uj8up3kDr9I2I5HOFZrtk2FZV3YNh28Zskqr+VggulkdMxdcGVP4M1Oe+D/fK3jaSqbdOLx6Cm E+F0XuEFsPKRtT6NuigZblL7DTnLa54exuJ4XA5og5tI/qi6OZEIwqX5QhHPA72XiYu7nK5TmEkpk E2Ykm/cXh0rIyLPR9XhPfk0uTGLtT0WZ0yy6tmp/fMbGkQcMmfbMreR3zibxxiPQXNg7d64mebEul EYs5jZ8KSe9cVFIGO/v48N5qu8pTH8SjIUa8KKXu7cxZINDIXJl694OGjeDe1VlcV2ftDh/wP9dNu kGYacgmru4Ly/ny5VdBw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz51-0000000H7UU-2lFL; Fri, 01 Mar 2024 09:21:23 +0000 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz4y-0000000H7Sb-1ZKZ for linux-arm-kernel@lists.infradead.org; Fri, 01 Mar 2024 09:21:21 +0000 Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-33d61e39912so854981f8f.3 for ; Fri, 01 Mar 2024 01:21:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709284878; x=1709889678; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=roMginZSvi1o0fjo+EUJOo1pu0LVGawrxcxtyn24YPc=; b=H64wa1m3FodKj2Y1O47ux4KB9vIHYQdCwBw/PLBbIjK3jsY7LT1PN+2tFYRgfhON/2 zUfDHuYE1hkztGgSwHF7o/ywY6rlMcHGnJdNJBNe9NBau/z9BP9aVwxqzrJaV7yhs1XM c9mJJ9fI8UyG5fi4FlP1dO5ylla0aKli4nL7+l+K9IzKbPTnoRfbgSTO2y3gwgS8Cmkj Jcavkd19QSPmaUw8kXH1RPG1gPh4hbF8oSF3b7snGwZAFzc5+PR8s8Eq1/szUCWLAhs+ 1iOng/fLdap3CNNVOGqKkIl86U0IMOA5dCmgzkU7mxdHtL/7Qnrg59o7TqnLdgqC1w0e byRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709284878; x=1709889678; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=roMginZSvi1o0fjo+EUJOo1pu0LVGawrxcxtyn24YPc=; b=dflz5sfJKUKZPnYRSD3F3W1jQJylof5z4UJ8kNUeZrHDR8WfFYzGk+/A7qRSXig8GF N0nLLM1r/1jVdNe1HklzY2OZ555dSqWuGR+9apkjHYRpJhLXcWuYH1r/yOv+woy7OTSH qncT1oNhr1mT2m58kn+yYTohy4tHKI/JuQkeffIRV7dKYQZAj0a93vjgMb2KRlpF66Om D2MIC8AC5nlzEnuUxLO16SWzcM8vXBs21o9JFkzuWrem/Iy48j0R4mYTV9AwJgN+UHgR oDAkZai9OxD0YdLeVRLKKOg5KsR40pfQnQd4B3SZvR6dseNIyD1unGyOIOxFyKlMOxsr cBUw== X-Forwarded-Encrypted: i=1; AJvYcCUZro/MU5ZTzPV/04xVATfkP30FDc8Im4sqrVGI0/WaXkijlUX36Hdg7DSYq7vGUf8aXn+GAHhLEis11KGcJ9uLALvR3EzsAiGSEZbEJv1BbyRl6/I= X-Gm-Message-State: AOJu0Yym7ybE+GCnrSVwQ28FX4zgVkeUiJlED3haDUrSSCIN1VXwbgUH 29jrV+0BF23sy9homIKxUIHGoob9f59Vh1Dk2S8NlkA/YGdBc3oX6rb4K2MCe57koJc+Cdo+4sQ 4 X-Google-Smtp-Source: AGHT+IHE36Cx0pzkf7/PwROG0sFnOnPILtKg/5tjC2XGRavDiE3m4OmzpuOVj7dxMVbSTb8/7tKiwA== X-Received: by 2002:adf:f24e:0:b0:33d:c5c5:9bc6 with SMTP id b14-20020adff24e000000b0033dc5c59bc6mr893274wrp.54.1709284878472; Fri, 01 Mar 2024 01:21:18 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id i6-20020adfb646000000b0033e033898c5sm4044660wre.20.2024.03.01.01.21.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 01:21:18 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 6/9] mm: Use common huge_ptep_get_and_clear() function for riscv/arm64 Date: Fri, 1 Mar 2024 10:14:52 +0100 Message-Id: <20240301091455.246686-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240301091455.246686-1-alexghiti@rivosinc.com> References: <20240301091455.246686-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240301_012120_458653_8D9B6472 X-CRM114-Status: GOOD ( 21.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Note that get_clear_contig() function is duplicated in the generic and the arm64 code because it is still used by some arm64 functions that will, in the next commits, be moved to the generic code. Once all have been moved, the arm64 version will be removed. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 14 +++++++++- arch/arm64/mm/hugetlbpage.c | 19 ++----------- arch/riscv/include/asm/pgtable.h | 4 ++- arch/riscv/mm/hugetlbpage.c | 21 ++------------ mm/contpte.c | 48 ++++++++++++++++++++++++++++++-- 5 files changed, 66 insertions(+), 40 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 437e9638b2b9..b5caf7a9d03f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1131,11 +1131,23 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); -static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, +int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize); + +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { int contig_ptes = 0; + /* + * If the size is not passed, we need to go through the page table to + * find out the number of contiguous ptes. + */ + if (size == 0) + return find_num_contig(mm, addr, ptep, pgsize); + *pgsize = size; switch (size) { diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 09b55bac8c7c..a2d33cbc7da5 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -100,8 +100,8 @@ int pud_huge(pud_t pud) #endif } -static int find_num_contig(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, size_t *pgsize) +int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize) { pgd_t *pgdp = pgd_offset(mm, addr); p4d_t *p4dp; @@ -284,21 +284,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - int ncontig; - size_t pgsize; - pte_t orig_pte = ptep_get(ptep); - - if (!pte_cont(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - - return get_clear_contig(mm, addr, ptep, pgsize, ncontig); -} - /* * huge_ptep_set_access_flags will update access flags (dirty, accesssed) * and write permission. diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 03f8ced8b26a..1cb877f0d0ce 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -530,7 +530,9 @@ static inline void __set_pte_at(pte_t *ptep, pte_t pteval) } #ifdef CONFIG_RISCV_ISA_SVNAPOT -static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { unsigned long hugepage_shift; diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 01b1403dd4cb..4c441664db8a 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -187,7 +187,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); @@ -202,21 +202,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return true; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep) -{ - pte_t orig_pte = ptep_get(ptep); - int pte_num; - - if (!pte_napot(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); - - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); - - return get_clear_contig(mm, addr, ptep, pte_num); -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -231,7 +216,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, return; } - pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + pte_num = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); @@ -249,7 +234,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_clear_flush(vma, addr, ptep); - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } diff --git a/mm/contpte.c b/mm/contpte.c index 5200c234404d..8d2ab391e0d8 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -20,6 +20,7 @@ * huge_ptep_get() * set_huge_pte_at() * huge_pte_clear() + * huge_ptep_get_and_clear() */ pte_t huge_ptep_get(pte_t *ptep) @@ -31,7 +32,7 @@ pte_t huge_ptep_get(pte_t *ptep) if (!pte_present(orig_pte) || !pte_cont(orig_pte)) return orig_pte; - ncontig = arch_contpte_get_num_contig(ptep, + ncontig = arch_contpte_get_num_contig(NULL, 0, ptep, page_size(pte_page(orig_pte)), &pgsize); @@ -85,7 +86,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, int i; int ncontig; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) @@ -109,8 +110,49 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, int i, ncontig; size_t pgsize; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) pte_clear(mm, addr, ptep); } + +static pte_t get_clear_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + pte_t orig_pte = ptep_get(ptep); + unsigned long i; + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { + pte_t pte = ptep_get_and_clear(mm, addr, ptep); + + /* + * If HW_AFDBM (arm64) or svadu (riscv) is enabled, then the HW + * could turn on the dirty or accessed bit for any page in the + * set, so check them all. + */ + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + return orig_pte; +} + +pte_t huge_ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + int ncontig; + size_t pgsize; + pte_t orig_pte = ptep_get(ptep); + + if (!pte_cont(orig_pte)) + return ptep_get_and_clear(mm, addr, ptep); + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + return get_clear_contig(mm, addr, ptep, pgsize, ncontig); +} From patchwork Fri Mar 1 09:14:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13578209 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9AF0CC5475B for ; Fri, 1 Mar 2024 09:22:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CJxXzgZ3PuTsnEtHWPiCZgmxOHT3qzNoZg6tP1tzpxM=; b=IEozXxaplNtDzg 1DDfBk7dnOiT5TlICm9Rm4ZJR4fCZ9XzasCriZfEe1h1brD2uhoqu4NyR4SeYZMSJIZ/qynt6L3Al nHl/ac0DNMFtzu9fasSr5h92lUeef+hUeGn/Orhc5gxWmfmxSekRRqaev09e7JMZsiVkRYbnW3bMA 1yzyEX9UF5DSYXAbUAHMtyPh+pgehLes4HxINRpIaDEgMVaMKTQ7uXUiEPnrkxorYpl1RA/dz2xoZ gKn4cPPsACcIMKo7Vv/hyCX8/ZwTu7jX0lwwUjNAeFwI3IpcFNBMVyxwAMs6groP5xjE5UvxfjjD5 yaDf4+dnYjDo4FVKQiGw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz6M-0000000H84s-2HYx; Fri, 01 Mar 2024 09:22:47 +0000 Received: from mail-wr1-x436.google.com ([2a00:1450:4864:20::436]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz5x-0000000H7wd-1gFp for linux-arm-kernel@lists.infradead.org; Fri, 01 Mar 2024 09:22:31 +0000 Received: by mail-wr1-x436.google.com with SMTP id ffacd0b85a97d-33d90dfe73cso1005584f8f.0 for ; Fri, 01 Mar 2024 01:22:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709284939; x=1709889739; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fHzJQXP5BEIeqfOJsZUcjaIxOTpiaXVWcZpI+r0iAkE=; b=PkKjFO/EA83WbpzCUwN5bV51Ybfnnl7njjJ29NNqei0weZQZRhgQDNC3InOjYufRSV NWZlPoezk9kmEcHcbKBrdmfvU4wwgTyGUD0OCorF+P6ZxRB88gu5ixc1bbSus0JY4s8W o81fTW7oNFPU8mAJF1X6dcf7ay52Y4MrB5IjFAGCpbdTZjlCN4QAybEAS2NcFcqWYcd4 LqbpnwntD0xmOBX57FkhBevpVyJiH5BJjbs5WGRNNcm2WXD4DPkOOZpHe2tFd03bmFW5 9nqRvOzquY8rq0BBhKMZTRGeyCWwanf/LW6L6YAeETiLtFQNSgUbz8YFewwkSZmDKEMb ynKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709284939; x=1709889739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fHzJQXP5BEIeqfOJsZUcjaIxOTpiaXVWcZpI+r0iAkE=; b=babcpXFb/BzuQOdXFG4IYhJvuSmj/p81c8rea9DMQZLnvlDgW23JAmjEZ1iecnnhvj JFFG9lrx5FS8Hu5pZlmhUw5flmY5fYAs191mzrYCWb78qrK7bVEZbfuqC1bleKP4uHal 2QIiYwZdua+fB93lhnj7vorEP+LaNn2VTF0nVGt67vXsTjOO8BUJl4Dkz2tA0HydZJLC 537g648FxcAC3gtg9aX504A9nME+9KIEvAv1AW3W2Eox31F4TyBt7zuspSXu0KKc3Bme sZvLiTYSp26Q8Ll7pDlDP/RshyGYtFrHg9/ed9KDzb9UekU7/SZyNflf1hy6RnSJTgGf qPVA== X-Forwarded-Encrypted: i=1; AJvYcCUYgwb3dz/w/HnIOuE+gTZLdezxkAQWU7RMJmHnys9YK5nRIK4Hs398adsAMl3f3zuoFc1UV0MCs1DkpXff1maOn61GHk+xA2BEMRiApwNaWHQzmTE= X-Gm-Message-State: AOJu0Yyim+7oGkJ1N4P4ib42PZ+7oVPiyEcf8E2Rp2Q5wT50YvDKwMZl /dZhZ99ddJ0bQQbDGrzlMcFEZLMPrs7Ky8FimQpKVzvcvDiJDliwsaKb0Ru6bNE= X-Google-Smtp-Source: AGHT+IETq5VRJHPuwhCoVEqwX4u/EPRIvAhXp1XnZCRkR4LhBkbxP9zDwDTrac3tIekTrNx6Id8PTw== X-Received: by 2002:adf:f84c:0:b0:33d:39db:a0f8 with SMTP id d12-20020adff84c000000b0033d39dba0f8mr1044349wrq.7.1709284939471; Fri, 01 Mar 2024 01:22:19 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id e13-20020a5d594d000000b0033e12b2e567sm3302518wri.35.2024.03.01.01.22.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 01:22:19 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 7/9] mm: Use common huge_ptep_set_access_flags() function for riscv/arm64 Date: Fri, 1 Mar 2024 10:14:53 +0100 Message-Id: <20240301091455.246686-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240301091455.246686-1-alexghiti@rivosinc.com> References: <20240301091455.246686-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240301_012222_604721_32BAF8CD X-CRM114-Status: GOOD ( 20.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Both architectures have almost the same implementation: __cont_access_flags_changed() is also correct on riscv and brings the same benefits (ie don't do anything if the flags are unchanged). As in the previous commit, get_clear_contig_flush() is duplicated in both the arch and the generic codes, it will be removed from the arch code when the last reference there gets moved to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/mm/hugetlbpage.c | 65 -------------------------------- arch/riscv/mm/hugetlbpage.c | 29 --------------- mm/contpte.c | 74 +++++++++++++++++++++++++++++++++++++ 3 files changed, 74 insertions(+), 94 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index a2d33cbc7da5..b8abbd5ace4f 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -284,71 +284,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -/* - * huge_ptep_set_access_flags will update access flags (dirty, accesssed) - * and write permission. - * - * For a contiguous huge pte range we need to check whether or not write - * permission has to change only on the first pte in the set. Then for - * all the contiguous ptes we need to check whether or not there is a - * discrepancy between dirty or young. - */ -static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig) -{ - int i; - - if (pte_write(pte) != pte_write(ptep_get(ptep))) - return 1; - - for (i = 0; i < ncontig; i++) { - pte_t orig_pte = ptep_get(ptep + i); - - if (pte_dirty(pte) != pte_dirty(orig_pte)) - return 1; - - if (pte_young(pte) != pte_young(orig_pte)) - return 1; - } - - return 0; -} - -int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t pte, int dirty) -{ - int ncontig, i; - size_t pgsize = 0; - unsigned long pfn = pte_pfn(pte), dpfn; - struct mm_struct *mm = vma->vm_mm; - pgprot_t hugeprot; - pte_t orig_pte; - - if (!pte_cont(pte)) - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - dpfn = pgsize >> PAGE_SHIFT; - - if (!__cont_access_flags_changed(ptep, pte, ncontig)) - return 0; - - orig_pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); - - /* Make sure we don't lose the dirty or young state */ - if (pte_dirty(orig_pte)) - pte = pte_mkdirty(pte); - - if (pte_young(orig_pte)) - pte = pte_mkyoung(pte); - - hugeprot = pte_pgprot(pte); - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); - - return 1; -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 4c441664db8a..fdb633844d4b 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -173,35 +173,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, - pte_t *ptep, - pte_t pte, - int dirty) -{ - struct mm_struct *mm = vma->vm_mm; - size_t pgsize; - pte_t orig_pte; - int pte_num; - - if (!pte_napot(pte)) - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - - pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); - - orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); - - if (pte_dirty(orig_pte)) - pte = pte_mkdirty(pte); - - if (pte_young(orig_pte)) - pte = pte_mkyoung(pte); - - set_ptes(mm, addr, ptep, pte, pte_num); - - return true; -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) diff --git a/mm/contpte.c b/mm/contpte.c index 8d2ab391e0d8..3b65c49c6d4b 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -21,6 +21,7 @@ * set_huge_pte_at() * huge_pte_clear() * huge_ptep_get_and_clear() + * huge_ptep_set_access_flags() */ pte_t huge_ptep_get(pte_t *ptep) @@ -156,3 +157,76 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, return get_clear_contig(mm, addr, ptep, pgsize, ncontig); } + +/* + * huge_ptep_set_access_flags will update access flags (dirty, accesssed) + * and write permission. + * + * For a contiguous huge pte range we need to check whether or not write + * permission has to change only on the first pte in the set. Then for + * all the contiguous ptes we need to check whether or not there is a + * discrepancy between dirty or young. + */ +static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig) +{ + int i; + + if (pte_write(pte) != pte_write(ptep_get(ptep))) + return 1; + + for (i = 0; i < ncontig; i++) { + pte_t orig_pte = ptep_get(ptep + i); + + if (pte_dirty(pte) != pte_dirty(orig_pte)) + return 1; + + if (pte_young(pte) != pte_young(orig_pte)) + return 1; + } + + return 0; +} + +static pte_t get_clear_contig_flush(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + pte_t orig_pte = get_clear_contig(mm, addr, ptep, pgsize, ncontig); + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + + flush_tlb_range(&vma, addr, addr + (pgsize * ncontig)); + return orig_pte; +} + +int huge_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, int dirty) +{ + int ncontig; + size_t pgsize = 0; + struct mm_struct *mm = vma->vm_mm; + pte_t orig_pte; + + if (!pte_cont(pte)) + return ptep_set_access_flags(vma, addr, ptep, pte, dirty); + + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); + + if (!__cont_access_flags_changed(ptep, pte, ncontig)) + return 0; + + orig_pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); + + /* Make sure we don't lose the dirty or young state */ + if (pte_dirty(orig_pte)) + pte = pte_mkdirty(pte); + + if (pte_young(orig_pte)) + pte = pte_mkyoung(pte); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); + + return 1; +} From patchwork Fri Mar 1 09:14:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13578210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 278D4C5478C for ; Fri, 1 Mar 2024 09:23:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wHkjHSzOLK0ecA7g4toscZ18ORenETBRMW0twTDvz+o=; b=gSpzt8p76kJuah VZMJozz8rfMG7gjDkZaVsiaYTTDlUEF+QJbPIR1II8Er8h8zi7x9HTpGn5Ok5bVF/bgn4iUAmfVhG Pnv+KVajFPIP/6Whu22wf6HJV948RXBnr0t5pfHXtQItlWw7exidH+NOr+19w0Yn4JQiVU7goMRQM /9oJLnUww+wZ6AFip7aENNX4S9gxDtH4dRYPOGbwVYBiaZDWixgYrkHTJyhABVGONDwGAf5DqeEpr dm5SBFUdyVuq1y/jJlcFTp7v/Da9kaFmzNLbghwo7P1bUoIgOvbBOkdnmslXagl3PEJ3BshkyQrNu V5R+29aKOHymzUPVQGSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz6y-0000000H8P3-12CR; Fri, 01 Mar 2024 09:23:24 +0000 Received: from mail-lj1-x22c.google.com ([2a00:1450:4864:20::22c]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz6w-0000000H8Na-0nve for linux-arm-kernel@lists.infradead.org; Fri, 01 Mar 2024 09:23:23 +0000 Received: by mail-lj1-x22c.google.com with SMTP id 38308e7fff4ca-2d22b8801b9so23166131fa.0 for ; Fri, 01 Mar 2024 01:23:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709285000; x=1709889800; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+Bwm42ASg+Pn4jchqsA5u8tprYh0PuExgujdSk24Zd8=; b=cXT22CyG/d8SqYLbha2zxQpDcO5gf7kppO4zPTcN5pfPB+lr0KUtXU5gOPPICVmdmz Kv1bHfhLhX7W1wW0FerSMd6H7QlZjAheMMbMmt3m5wdANMCYHhFm4SmwJ1Rp29hbWigG E6xSMrxzralVeI12KRBHH5Tn1Jf/mr5qod01N4d4Prlz7Z+Vq0D32U/6o/5xyQvAySCE fZ3Rhhj/c5jJ1VHdyCV0hRMsp87AAyeUe+CJNP/fI4C0SoctWRnHNe+8kTQ/hYyuVd60 2tIypuEqUoNL/dU376DJ3C3uxh0ml/6eeEkK4xQqe65Z5zNkhaS4VLCxfhDZjKuZ1Z1w WavQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709285000; x=1709889800; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+Bwm42ASg+Pn4jchqsA5u8tprYh0PuExgujdSk24Zd8=; b=qCS4imjJd91FVClun0m0tRfxQFHTiKRvOjPnMlbEMXbT7YTXHuS8g4STws/GIUrqek dEXhYcr7P+ziyd4FwCRN+6YlB9UBacoIY17JuJ1IpZM2UU+jE2WomKzZ2JzTgzNbz36l otxUnOaxLKYhBKlXcUWr2ngjbWMfzzgF2JGuPyGX4bKvIKSwheouKbLkVivNBKKa8oVB SrIQRySZWz2BPLG/X2pjzgE7wl1ZcSaz5Qu8lRghoZtbuhHQ+CNFWM5jBvMLbS7YDym5 Amx6EQXOSea2dAtHPY7J6+TfZll2GmKxiWlk4AmXsqdNr4tIvALrOUbT29rtkdZMPnIs vaaQ== X-Forwarded-Encrypted: i=1; AJvYcCUlAO8F6RExRH9XalY/UC+iyh6CfZVT7jzQLiuVMZCMJ4Xt0JG7odkZgauGmrgX5leWYSV8ywdr6DuhSZS4sJrBzGJsOrHvbkHF0pviB55qw/OiQ7Y= X-Gm-Message-State: AOJu0YyNRWBNgl512cpkIbTZX+pH9HZJBbOLvdCQG7YxeXLsZuAZMD4B U99osr1ODZAK9UToILthjxbg21XfJMHEQpDi1nFuhBZapPk5H7yx66K61Vf4uLo= X-Google-Smtp-Source: AGHT+IFkOrueHwu+dCituNGudpmJQSWFJOG3RRqM9+lTLTigiW3MgJH56JMq1RpUEmcE9Sa98B2xOQ== X-Received: by 2002:a2e:a4cb:0:b0:2d3:365e:9d02 with SMTP id p11-20020a2ea4cb000000b002d3365e9d02mr635537ljm.28.1709285000535; Fri, 01 Mar 2024 01:23:20 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id l33-20020a05600c1d2100b00412ca88537dsm374810wms.0.2024.03.01.01.23.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 01:23:20 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 8/9] mm: Use common huge_ptep_set_wrprotect() function for riscv/arm64 Date: Fri, 1 Mar 2024 10:14:54 +0100 Message-Id: <20240301091455.246686-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240301091455.246686-1-alexghiti@rivosinc.com> References: <20240301091455.246686-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240301_012322_276370_B9E2C92E X-CRM114-Status: GOOD ( 13.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/mm/hugetlbpage.c | 27 --------------------------- arch/riscv/mm/hugetlbpage.c | 22 ---------------------- mm/contpte.c | 21 +++++++++++++++++++++ 3 files changed, 21 insertions(+), 49 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index b8abbd5ace4f..d6ddaf282a94 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -284,33 +284,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -void huge_ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - unsigned long pfn, dpfn; - pgprot_t hugeprot; - int ncontig, i; - size_t pgsize; - pte_t pte; - - if (!pte_cont(READ_ONCE(*ptep))) { - ptep_set_wrprotect(mm, addr, ptep); - return; - } - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - dpfn = pgsize >> PAGE_SHIFT; - - pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); - pte = pte_wrprotect(pte); - - hugeprot = pte_pgprot(pte); - pfn = pte_pfn(pte); - - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); -} - pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index fdb633844d4b..e6cbb6fb2904 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -173,28 +173,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -void huge_ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep) -{ - pte_t pte = ptep_get(ptep); - size_t pgsize; - pte_t orig_pte; - int pte_num; - - if (!pte_napot(pte)) { - ptep_set_wrprotect(mm, addr, ptep); - return; - } - - pte_num = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); - - orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); - orig_pte = pte_wrprotect(orig_pte); - - set_ptes(mm, addr, ptep, orig_pte, pte_num); -} - pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) diff --git a/mm/contpte.c b/mm/contpte.c index 3b65c49c6d4b..f7f26d2cfa23 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -22,6 +22,7 @@ * huge_pte_clear() * huge_ptep_get_and_clear() * huge_ptep_set_access_flags() + * huge_ptep_set_wrprotect() */ pte_t huge_ptep_get(pte_t *ptep) @@ -230,3 +231,23 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return 1; } + +void huge_ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + int ncontig; + size_t pgsize; + pte_t pte; + + if (!pte_cont(ptep_get(ptep))) { + ptep_set_wrprotect(mm, addr, ptep); + return; + } + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); + pte = pte_wrprotect(pte); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); +} From patchwork Fri Mar 1 09:14:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13578217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EC1EAC5478C for ; Fri, 1 Mar 2024 09:24:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UHIDziMeprGDGqlCCT4OwEdnw4TQIRr7uyGH2f0qhos=; b=ulFkLJcM2/3glB dP3HR89K4ibhOFzZo338NbdtIepZjZx3/jY5WdgqTKsYch2k2j12Sgt6Fm/+LCSspFOQ5pWPyntZh bQpdXYopNZJIDLu7EdVRfrmEbKGQKfRkCqpl3V33S5SQR3Gnod1o+KudNCh165ItlrEU/4aLUXpbg ZOwv5AsktiZ7et+FyZoJ8753r1eeGCkFqRQp55B3fVB1Q7smwLpXv3FjXnYKeQ/MGHv6qs1t9dbY1 ENHA/dv5n/w71i2Q5RAA5RgPEvblXe2sBzWwj8daRj+QOLiIMZ3znJBGY0zogy9ghvNTZkJa9O4bU pOJNTZT2lI4pCumGaeEA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz8E-0000000H8zd-23H1; Fri, 01 Mar 2024 09:24:45 +0000 Received: from mail-lj1-x233.google.com ([2a00:1450:4864:20::233]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rfz80-0000000H8qp-3euY for linux-arm-kernel@lists.infradead.org; Fri, 01 Mar 2024 09:24:35 +0000 Received: by mail-lj1-x233.google.com with SMTP id 38308e7fff4ca-2d21cdbc85bso19737181fa.2 for ; Fri, 01 Mar 2024 01:24:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1709285066; x=1709889866; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K12kuYfal65hYkuTm7CnsK9zS+VdU7mNn9HZDxR8IYY=; b=G+tTGKTQZarcotf0nrcp4ANhRn44YYHs4byoBEfZMWvy43AZxLNfaFcdzEsGkopsdy JSoBTvsj2WU07MMEdfrkqR2qjQtRmeRIO4fHInXEKnwmZbp8mU4phU2vsPV7N+PShIB6 vstKHVvrX/CKTVw5i+t925KAi8Qcr4wpd5+Xf+Dn9lamLryoUXJ+xIegPWxRy/tJGZu/ BRJbAbDUDC7mgXZV6fh5yxL/vGmchhS+HqQn2kQST2gBq85z2aCvgIN0R3cL1eR6Ft7A kdLV2zYms3VDIMO0T/fnDFWvzco3PSvX9vbrXBHapACx6+fo6m9sGdFhc3kNF2fxPir2 Hh5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709285066; x=1709889866; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K12kuYfal65hYkuTm7CnsK9zS+VdU7mNn9HZDxR8IYY=; b=vhy2L0TC4fY5Nv4cdqmKsVHiJBWt+QsZVK7J/rgnQ+B4VNmnOS7d0U4j8cAS3p7C9G XP4nRQ1ZoYePBHNKWWHdyzda4s+zcN7NVKsAm5S1mE4cV6iz7zDtW1wO6xzjd1xlyZZW 3cbtOocazZHcLsWSp1EiR2rJyRq/1dBmURRHyA01xg09Dod+haEGyCnvgxogoH9BTDhP MSdTIQqPSekaSOSc92rvVHAwt8Ubpc96KEsbfkgdlbDbfEIyPWPETDioWkcebaZtF8Tg M5HgdTchc5XnVFkznsVyamg/x+hB3F4oMx0mGa72S8zIIKJy40OTFBSgBvytp1gSdlf/ c6Ug== X-Forwarded-Encrypted: i=1; AJvYcCU/hL6ARxuvlbhuzU7ebwTP8Ji8zS621HpTfiFL06W5wUxu2mWNEOAUaGl50R+WZdSsL3AnobaZKIlNxVEctYlrTR6t8ummBP9/Bz/1BUw78DdBZqg= X-Gm-Message-State: AOJu0YwGIBZXQKYaeUcSzQiMUxJIei9oaFDuklfRpBpMOpOQB2DGKNyc PRabHCFsOB1dtWblQsCDpRMreW9bhmgJ+mjpYRJIjtm6tR59cnGQpODk2akMbT8= X-Google-Smtp-Source: AGHT+IFt+kCs8abRI8MZlthQ5rKNhpeiYCQtIBul4mlcyfPmRwi1Z8kSKZm+NO/gjQvAypId4axWxw== X-Received: by 2002:a2e:9909:0:b0:2d2:92a2:9a84 with SMTP id v9-20020a2e9909000000b002d292a29a84mr731669lji.43.1709285061585; Fri, 01 Mar 2024 01:24:21 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id w15-20020a05600c474f00b00412ca1dc2e4sm649267wmo.7.2024.03.01.01.24.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Mar 2024 01:24:21 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 9/9] mm: Use common huge_ptep_clear_flush() function for riscv/arm64 Date: Fri, 1 Mar 2024 10:14:55 +0100 Message-Id: <20240301091455.246686-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240301091455.246686-1-alexghiti@rivosinc.com> References: <20240301091455.246686-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240301_012429_143265_21E1DBBA X-CRM114-Status: GOOD ( 17.14 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/mm/hugetlbpage.c | 61 ------------------------------------- arch/riscv/mm/hugetlbpage.c | 51 ------------------------------- mm/contpte.c | 15 +++++++++ 3 files changed, 15 insertions(+), 112 deletions(-) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index d6ddaf282a94..8a273b9816d7 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -119,53 +119,6 @@ int find_num_contig(struct mm_struct *mm, unsigned long addr, return CONT_PTES; } -/* - * Changing some bits of contiguous entries requires us to follow a - * Break-Before-Make approach, breaking the whole contiguous set - * before we can change any entries. See ARM DDI 0487A.k_iss10775, - * "Misprogramming of the Contiguous bit", page D4-1762. - * - * This helper performs the break step. - */ -static pte_t get_clear_contig(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - pte_t orig_pte = ptep_get(ptep); - unsigned long i; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { - pte_t pte = ptep_get_and_clear(mm, addr, ptep); - - /* - * If HW_AFDBM is enabled, then the HW could turn on - * the dirty or accessed bit for any page in the set, - * so check them all. - */ - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - return orig_pte; -} - -static pte_t get_clear_contig_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - pte_t orig_pte = get_clear_contig(mm, addr, ptep, pgsize, ncontig); - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - - flush_tlb_range(&vma, addr, addr + (pgsize * ncontig)); - return orig_pte; -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz) { @@ -284,20 +237,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - struct mm_struct *mm = vma->vm_mm; - size_t pgsize; - int ncontig; - - if (!pte_cont(READ_ONCE(*ptep))) - return ptep_clear_flush(vma, addr, ptep); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); -} - static int __init hugetlbpage_init(void) { if (pud_sect_supported()) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index e6cbb6fb2904..caf1db6f8f20 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -121,42 +121,6 @@ unsigned long hugetlb_mask_last_page(struct hstate *h) return 0UL; } -static pte_t get_clear_contig(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pte_num) -{ - pte_t orig_pte = ptep_get(ptep); - unsigned long i; - - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) { - pte_t pte = ptep_get_and_clear(mm, addr, ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - - return orig_pte; -} - -static pte_t get_clear_contig_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pte_num) -{ - pte_t orig_pte = get_clear_contig(mm, addr, ptep, pte_num); - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - bool valid = !pte_none(orig_pte); - - if (valid) - flush_tlb_range(&vma, addr, addr + (PAGE_SIZE * pte_num)); - - return orig_pte; -} - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) { unsigned long order; @@ -173,21 +137,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, - unsigned long addr, - pte_t *ptep) -{ - pte_t pte = ptep_get(ptep); - int pte_num; - - if (!pte_napot(pte)) - return ptep_clear_flush(vma, addr, ptep); - - pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); - - return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); -} - static bool is_napot_size(unsigned long size) { unsigned long order; diff --git a/mm/contpte.c b/mm/contpte.c index f7f26d2cfa23..445e5ebe46b4 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -23,6 +23,7 @@ * huge_ptep_get_and_clear() * huge_ptep_set_access_flags() * huge_ptep_set_wrprotect() + * huge_ptep_clear_flush() */ pte_t huge_ptep_get(pte_t *ptep) @@ -251,3 +252,17 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, set_contptes(mm, addr, ptep, pte, ncontig, pgsize); } + +pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + struct mm_struct *mm = vma->vm_mm; + size_t pgsize; + int ncontig; + + if (!pte_cont(ptep_get(ptep))) + return ptep_clear_flush(vma, addr, ptep); + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); +}