From patchwork Wed Aug 2 15:13:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13338370 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 836E3C04E69 for ; Wed, 2 Aug 2023 15:15:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3D582801B1; Wed, 2 Aug 2023 11:14:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AC7402801AA; Wed, 2 Aug 2023 11:14:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 98DCA2801B1; Wed, 2 Aug 2023 11:14:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 89E262801AA for ; Wed, 2 Aug 2023 11:14:37 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 674101A0712 for ; Wed, 2 Aug 2023 15:14:37 +0000 (UTC) X-FDA: 81079511394.13.A217F3E Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf21.hostedemail.com (Postfix) with ESMTP id 7C4C71C0007 for ; Wed, 2 Aug 2023 15:14:35 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="Ha9s/sKN"; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1690989275; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wBzUOAudwvibgR05xcILaDI1lHt6uM+gLVFXP8bsuc0=; b=XvRXhX430W5kQttiY1gD+5/n2CSJSaXaXWm21duPSW+F2Ut3m4pSymcIcEbFmRhMvUxOHQ Psxk9DPHglRj4FGmDAkZcYmCeQTdFUBtKskH1UjAr1sqEcw75GkqYmqMXC8zl6WZI31/pG /7TabRM59umx4HPMCCr9ab3tzWa/HUk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1690989275; a=rsa-sha256; cv=none; b=Uug559UPOx2nidsbK155qYg4I9Wdk5gEHq9ZfmZpAYItSKO5BfPGz2+6kAGw3o0IVNWgBO Co8WkXN4+QYDjCDtjy9Vdgs7zy18Psv+Da0NrGM1658e/a45+0cWTChXhCQVtzAkl+03qa dcqhcn9iWYtZ2Xxb1JUIk4DAA0vAi1A= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b="Ha9s/sKN"; spf=none (imf21.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=wBzUOAudwvibgR05xcILaDI1lHt6uM+gLVFXP8bsuc0=; b=Ha9s/sKNhdfeybdIYofHo+zqU+ ot8ZgcRg0OXHpTfLXK7qTZ+NFCab/7cCF/cAPpEv7EwGg0NVhtKlv8fxjnbEEnrV9RHLiHvYRTLVx sKmKQRJX6gkR36XaRIHbh6MMjX8PV5CP8uci9/fOC0QF8+vUc8adPFYjIwD4ccpBFnRTaxDRL0lY2 25G6tjp5LzEW8RBGuX20uqsI2hpOBlD0vRBc4n70B5Iqq/RJnjpiIstoVFEDu5/MUkD0ghh+fDtGi KdyynwzxrxNTdwR79IRw0WeUOQ0RWGwgXQrNVErQWXyR3bFnYu0BtiALQ5p2J4wGQ6keaOBySlAsV 2x50BKdA==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1qRDY9-00FfjV-D6; Wed, 02 Aug 2023 15:14:09 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Mike Rapoport , Huacai Chen , WANG Xuerui , loongarch@lists.linux.dev Subject: [PATCH v6 14/38] loongarch: Implement the new page table range API Date: Wed, 2 Aug 2023 16:13:42 +0100 Message-Id: <20230802151406.3735276-15-willy@infradead.org> X-Mailer: git-send-email 2.37.1 In-Reply-To: <20230802151406.3735276-1-willy@infradead.org> References: <20230802151406.3735276-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7C4C71C0007 X-Rspam-User: X-Stat-Signature: o7i93s3qdtx5sj89jou3yg5xazj1auyf X-Rspamd-Server: rspam03 X-HE-Tag: 1690989275-397447 X-HE-Meta: U2FsdGVkX1/bDzmE7Yk1Ii++raa6DqNjjOXqls/d0NYB7AaKGaom4ZYxNgKAkhEW0DzgvdzhGG4MUXghfvriQm7zFJuJFiAEKmI6QVq6Ej0CgcoC2pRwn6gauRDjtFePXCBPfHtRJtjMowJJ9flaVXzPOICqu6dzSRZl6yCixvg1/8h2wRGfqH571K5lS0xHeJ8qRXw4Ven1m0+Nr9degpuZ+VI6SieQ0DpBFTgfXRBKjpJ7Gb9UfuLmOwI/NMJcaFqrdvPmbt/qv1XWvajcl2Dw8xftb9mpvPW6UCO7yjl5uADwSlIuRTU5/CrMmMuKkGSeejvhCkdd9jQmqP2dDLAQe3wixFvsBjBj7MpM1Fs5SEwyPxN016IaQw3d8sdJaFQWrR8xaWQQYFAdxBpjJ0SXHOWwUXIUjulJqA/i3eEg8v0cYiwtMJ/oJuuMiBU2CkohLfxRuuM1zM6sw3xh3ULc6uFdD2AdwgSRzVvSUldoM7d1RU9Tzs+bKvlowYgfnOI/rO5x4kLBBh48a5avisFbYxD1JKzSpvdVYQma+eL2M/o+qwUn/NolKEEy/KLI1N7kYZVrc9LabdHXSc+F2UY/DhxSAMXV6KHYr5EndJht/xerTmqhtwRf2n3XoFknGo+sf5VsqP+xEUzVyea6rOsK3o8WJZ5DgNOXLPN9wic7yaxHegj3lyM193Z05Y3csg2qzYIHey8Ls6YFLwCJ4LMGYwDkIAY48EUkMc12ERv9sa4++YzrcAEhVM+c+ouLvt/tDRcT5p4d4TGsrQwm9EK3baD0P3FIiAibpiAqNrxSiSdQN3AT9VEHLZaqlcpQdqShGpE3DoT8u1+j/yHu98E7cKHENgSF1HCkrjlHJ2uj3mbJ/vgQ/hpq/GubJkHjdYjwcRMzBndzOrYMvuZmDveGMyh7c390G+yalamhPX9QSWmXPfY2Gr1jo0BO4nXnFq1X55z+sCjfDgwwJPU xyqXKmPf oMhM2sUOMWzilZwSBUpTrv/pWU9HrPUpCEchKHe7UA6LAu7nmETNWZS40txhrdC3R7m2PMsQdIPL5wTfQOrB2SmJhQpUGGcVqbmoQJ6tHV2s5qDdIlQLHFKhYGUkFW8JndR3mGFPC1uhdEqBSXrLTuACI6mm4bS7k1O8asrOv4qc9kZo8m15MgNPRbFsyqmw6UxYZC2zqN+4GoQykvJ8GkGT4kbpjeVoh6ZszVPPn1YnsAD8/oXSExIBLVA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add update_mmu_cache_range() and change _PFN_SHIFT to PFN_PTE_SHIFT. It would probably be more efficient to implement __update_tlb() by flushing the entire folio instead of calling __update_tlb() N times, but I'll leave that for someone who understands the architecture better. Signed-off-by: Matthew Wilcox (Oracle) Acked-by: Mike Rapoport (IBM) Cc: Huacai Chen Cc: WANG Xuerui Cc: loongarch@lists.linux.dev --- arch/loongarch/include/asm/cacheflush.h | 1 + arch/loongarch/include/asm/pgtable-bits.h | 4 +-- arch/loongarch/include/asm/pgtable.h | 33 ++++++++++++----------- arch/loongarch/mm/pgtable.c | 2 +- arch/loongarch/mm/tlb.c | 2 +- 5 files changed, 23 insertions(+), 19 deletions(-) diff --git a/arch/loongarch/include/asm/cacheflush.h b/arch/loongarch/include/asm/cacheflush.h index 0681788eb474..88a44da50a3b 100644 --- a/arch/loongarch/include/asm/cacheflush.h +++ b/arch/loongarch/include/asm/cacheflush.h @@ -47,6 +47,7 @@ void local_flush_icache_range(unsigned long start, unsigned long end); #define flush_cache_vmap(start, end) do { } while (0) #define flush_cache_vunmap(start, end) do { } while (0) #define flush_icache_page(vma, page) do { } while (0) +#define flush_icache_pages(vma, page) do { } while (0) #define flush_icache_user_page(vma, page, addr, len) do { } while (0) #define flush_dcache_page(page) do { } while (0) #define flush_dcache_mmap_lock(mapping) do { } while (0) diff --git a/arch/loongarch/include/asm/pgtable-bits.h b/arch/loongarch/include/asm/pgtable-bits.h index de46a6b1e9f1..35348d4c4209 100644 --- a/arch/loongarch/include/asm/pgtable-bits.h +++ b/arch/loongarch/include/asm/pgtable-bits.h @@ -50,12 +50,12 @@ #define _PAGE_NO_EXEC (_ULCAST_(1) << _PAGE_NO_EXEC_SHIFT) #define _PAGE_RPLV (_ULCAST_(1) << _PAGE_RPLV_SHIFT) #define _CACHE_MASK (_ULCAST_(3) << _CACHE_SHIFT) -#define _PFN_SHIFT (PAGE_SHIFT - 12 + _PAGE_PFN_SHIFT) +#define PFN_PTE_SHIFT (PAGE_SHIFT - 12 + _PAGE_PFN_SHIFT) #define _PAGE_USER (PLV_USER << _PAGE_PLV_SHIFT) #define _PAGE_KERN (PLV_KERN << _PAGE_PLV_SHIFT) -#define _PFN_MASK (~((_ULCAST_(1) << (_PFN_SHIFT)) - 1) & \ +#define _PFN_MASK (~((_ULCAST_(1) << (PFN_PTE_SHIFT)) - 1) & \ ((_ULCAST_(1) << (_PAGE_PFN_END_SHIFT)) - 1)) /* diff --git a/arch/loongarch/include/asm/pgtable.h b/arch/loongarch/include/asm/pgtable.h index 38afeb7dd58b..e7cf25e452c0 100644 --- a/arch/loongarch/include/asm/pgtable.h +++ b/arch/loongarch/include/asm/pgtable.h @@ -237,9 +237,9 @@ extern pmd_t mk_pmd(struct page *page, pgprot_t prot); extern void set_pmd_at(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp, pmd_t pmd); #define pte_page(x) pfn_to_page(pte_pfn(x)) -#define pte_pfn(x) ((unsigned long)(((x).pte & _PFN_MASK) >> _PFN_SHIFT)) -#define pfn_pte(pfn, prot) __pte(((pfn) << _PFN_SHIFT) | pgprot_val(prot)) -#define pfn_pmd(pfn, prot) __pmd(((pfn) << _PFN_SHIFT) | pgprot_val(prot)) +#define pte_pfn(x) ((unsigned long)(((x).pte & _PFN_MASK) >> PFN_PTE_SHIFT)) +#define pfn_pte(pfn, prot) __pte(((pfn) << PFN_PTE_SHIFT) | pgprot_val(prot)) +#define pfn_pmd(pfn, prot) __pmd(((pfn) << PFN_PTE_SHIFT) | pgprot_val(prot)) /* * Initialize a new pgd / pud / pmd table with invalid pointers. @@ -334,19 +334,13 @@ static inline void set_pte(pte_t *ptep, pte_t pteval) } } -static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pteval) -{ - set_pte(ptep, pteval); -} - static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { /* Preserve global status for the pair */ if (pte_val(*ptep_buddy(ptep)) & _PAGE_GLOBAL) - set_pte_at(mm, addr, ptep, __pte(_PAGE_GLOBAL)); + set_pte(ptep, __pte(_PAGE_GLOBAL)); else - set_pte_at(mm, addr, ptep, __pte(0)); + set_pte(ptep, __pte(0)); } #define PGD_T_LOG2 (__builtin_ffs(sizeof(pgd_t)) - 1) @@ -445,11 +439,20 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) extern void __update_tlb(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); -static inline void update_mmu_cache(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep) +static inline void update_mmu_cache_range(struct vm_fault *vmf, + struct vm_area_struct *vma, unsigned long address, + pte_t *ptep, unsigned int nr) { - __update_tlb(vma, address, ptep); + for (;;) { + __update_tlb(vma, address, ptep); + if (--nr == 0) + break; + address += PAGE_SIZE; + ptep++; + } } +#define update_mmu_cache(vma, addr, ptep) \ + update_mmu_cache_range(NULL, vma, addr, ptep, 1) #define __HAVE_ARCH_UPDATE_MMU_TLB #define update_mmu_tlb update_mmu_cache @@ -462,7 +465,7 @@ static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, static inline unsigned long pmd_pfn(pmd_t pmd) { - return (pmd_val(pmd) & _PFN_MASK) >> _PFN_SHIFT; + return (pmd_val(pmd) & _PFN_MASK) >> PFN_PTE_SHIFT; } #ifdef CONFIG_TRANSPARENT_HUGEPAGE diff --git a/arch/loongarch/mm/pgtable.c b/arch/loongarch/mm/pgtable.c index 36a6dc0148ae..1260cf30e3ee 100644 --- a/arch/loongarch/mm/pgtable.c +++ b/arch/loongarch/mm/pgtable.c @@ -107,7 +107,7 @@ pmd_t mk_pmd(struct page *page, pgprot_t prot) { pmd_t pmd; - pmd_val(pmd) = (page_to_pfn(page) << _PFN_SHIFT) | pgprot_val(prot); + pmd_val(pmd) = (page_to_pfn(page) << PFN_PTE_SHIFT) | pgprot_val(prot); return pmd; } diff --git a/arch/loongarch/mm/tlb.c b/arch/loongarch/mm/tlb.c index 00bb563e3c89..eb8572e201ea 100644 --- a/arch/loongarch/mm/tlb.c +++ b/arch/loongarch/mm/tlb.c @@ -252,7 +252,7 @@ static void output_pgtable_bits_defines(void) pr_define("_PAGE_WRITE_SHIFT %d\n", _PAGE_WRITE_SHIFT); pr_define("_PAGE_NO_READ_SHIFT %d\n", _PAGE_NO_READ_SHIFT); pr_define("_PAGE_NO_EXEC_SHIFT %d\n", _PAGE_NO_EXEC_SHIFT); - pr_define("_PFN_SHIFT %d\n", _PFN_SHIFT); + pr_define("PFN_PTE_SHIFT %d\n", PFN_PTE_SHIFT); pr_debug("\n"); }