From patchwork Tue Jul 28 09:45:33 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 6881971 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id D996CC05AC for ; Tue, 28 Jul 2015 09:54:53 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id C654D2042B for ; Tue, 28 Jul 2015 09:54:52 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 888752041C for ; Tue, 28 Jul 2015 09:54:51 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1ZK1Z1-0003Ut-Mo; Tue, 28 Jul 2015 09:52:47 +0000 Received: from foss.arm.com ([217.140.101.70]) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1ZK1SR-0004Vp-65 for linux-arm-kernel@lists.infradead.org; Tue, 28 Jul 2015 09:46:02 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 96C9675; Tue, 28 Jul 2015 02:45:46 -0700 (PDT) Received: from e104818-lin.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 788A13F318; Tue, 28 Jul 2015 02:45:36 -0700 (PDT) Date: Tue, 28 Jul 2015 10:45:33 +0100 From: Catalin Marinas To: Will Deacon Subject: Re: [PATCH 2/2] arm64: Use last level TLBI for user pte changes Message-ID: <20150728094533.GA16461@e104818-lin.cambridge.arm.com> References: <1437728396-5408-1-git-send-email-catalin.marinas@arm.com> <1437728396-5408-3-git-send-email-catalin.marinas@arm.com> <20150727174915.GM3358@arm.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20150727174915.GM3358@arm.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20150728_024559_683944_8BABDD99 X-CRM114-Status: GOOD ( 26.24 ) X-Spam-Score: -8.2 (--------) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "linux-arm-kernel@lists.infradead.org" , Steve Capper Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Spam-Status: No, score=-5.6 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Mon, Jul 27, 2015 at 06:49:15PM +0100, Will Deacon wrote: > On Fri, Jul 24, 2015 at 09:59:56AM +0100, Catalin Marinas wrote: > > The flush_tlb_page() function is used on user address ranges when PTEs > > (or PMDs/PUDs for huge pages) were changed (attributes or clearing). For > > such cases, it is more efficient to invalidate only the last level of > > the TLB with the "tlbi vale1is" instruction. > > > > In the TLB shoot-down case, the TLB caching of the intermediate page > > table levels (pmd, pud, pgd) is handled by __flush_tlb_pgtable() via the > > __(pte|pmd|pud)_free_tlb() functions and it is not deferred to > > tlb_finish_mmu() (as of commit 285994a62c80 - "arm64: Invalidate the TLB > > corresponding to intermediate page table levels"). The tlb_flush() > > function only needs to invalidate the TLB for the last level of page > > tables; a new arm64-specific __flush_tlb_page_range() function performs > > only the last level TLBI. > > [...] > > > diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h > > index 1643908eb5f3..48794ab79cc0 100644 > > --- a/arch/arm64/include/asm/tlbflush.h > > +++ b/arch/arm64/include/asm/tlbflush.h > > @@ -87,7 +87,7 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, > > ((unsigned long)ASID(vma->vm_mm) << 48); > > > > dsb(ishst); > > - asm("tlbi vae1is, %0" : : "r" (addr)); > > + asm("tlbi vale1is, %0" : : "r" (addr)); > > dsb(ish); > > } > > > > @@ -97,6 +97,26 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, > > */ > > #define MAX_TLB_RANGE (1024UL << PAGE_SHIFT) > > > > +static inline void __flush_tlb_page_range(struct vm_area_struct *vma, > > + unsigned long start, unsigned long end) > > +{ > > + unsigned long asid = (unsigned long)ASID(vma->vm_mm) << 48; > > + unsigned long addr; > > + > > + if ((end - start) > MAX_TLB_RANGE) { > > + flush_tlb_mm(vma->vm_mm); > > + return; > > + } > > + > > + start = asid | (start >> 12); > > + end = asid | (end >> 12); > > + > > + dsb(ishst); > > + for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) > > + asm("tlbi vale1is, %0" : : "r"(addr)); > > + dsb(ish); > > +} > > This is identical to __flush_tlb_range apart from the asm op. > > What happens if you add a "const bool leaf" parameter and stick a > conditional inside the loop? It looks better indeed (at the same code generated). The first patch remains unchanged and the second patch is below ("git am -c" should be able to detect the scissors line): -----8<-------------- From 8c95d0be9f25e2df6f53c16071494c4aee6aee87 Mon Sep 17 00:00:00 2001 From: Catalin Marinas Date: Wed, 18 Mar 2015 11:28:06 +0000 Subject: [PATCH v2] arm64: Use last level TLBI for user pte changes The flush_tlb_page() function is used on user address ranges when PTEs (or PMDs/PUDs for huge pages) were changed (attributes or clearing). For such cases, it is more efficient to invalidate only the last level of the TLB with the "tlbi vale1is" instruction. In the TLB shoot-down case, the TLB caching of the intermediate page table levels (pmd, pud, pgd) is handled by __flush_tlb_pgtable() via the __(pte|pmd|pud)_free_tlb() functions and it is not deferred to tlb_finish_mmu() (as of commit 285994a62c80 - "arm64: Invalidate the TLB corresponding to intermediate page table levels"). The tlb_flush() function only needs to invalidate the TLB for the last level of page tables; the __flush_tlb_range() function gains a fourth argument for last level TLBI. Signed-off-by: Catalin Marinas Cc: Will Deacon --- arch/arm64/include/asm/tlb.h | 7 ++++++- arch/arm64/include/asm/tlbflush.h | 21 ++++++++++++++++----- 2 files changed, 22 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 3a0242c7eb8d..d6e6b6660380 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -41,7 +41,12 @@ static inline void tlb_flush(struct mmu_gather *tlb) flush_tlb_mm(tlb->mm); } else { struct vm_area_struct vma = { .vm_mm = tlb->mm, }; - flush_tlb_range(&vma, tlb->start, tlb->end); + /* + * The intermediate page table levels are already handled by + * the __(pte|pmd|pud)_free_tlb() functions, so last level + * TLBI is sufficient here. + */ + __flush_tlb_range(&vma, tlb->start, tlb->end, true); } } diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 1643908eb5f3..c3a5f83e1276 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -87,7 +87,7 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, ((unsigned long)ASID(vma->vm_mm) << 48); dsb(ishst); - asm("tlbi vae1is, %0" : : "r" (addr)); + asm("tlbi vale1is, %0" : : "r" (addr)); dsb(ish); } @@ -97,8 +97,9 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, */ #define MAX_TLB_RANGE (1024UL << PAGE_SHIFT) -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) +static inline void __flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end, + bool last_level) { unsigned long asid = (unsigned long)ASID(vma->vm_mm) << 48; unsigned long addr; @@ -112,11 +113,21 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, end = asid | (end >> 12); dsb(ishst); - for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) - asm("tlbi vae1is, %0" : : "r"(addr)); + for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) { + if (last_level) + asm("tlbi vale1is, %0" : : "r"(addr)); + else + asm("tlbi vae1is, %0" : : "r"(addr)); + } dsb(ish); } +static inline void flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end) +{ + __flush_tlb_range(vma, start, end, false); +} + static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end) { unsigned long addr;