From patchwork Sat Mar 21 12:16:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhenyu Ye X-Patchwork-Id: 11451143 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CAF03913 for ; Sat, 21 Mar 2020 12:16:55 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A63D420722 for ; Sat, 21 Mar 2020 12:16:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="F/Qi6yd9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A63D420722 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Tp9XvE4Gmx6YipYIklqbOLtlQvJVXVCpwS73gvXxXR8=; b=F/Qi6yd9Xzn7us MLNjACQYDZxmOYc7gfuUXnRwKWnXB/Y5wLyh8wiISfB0AMYHbB5lZsIQaNAWjIXFN5Lvp0cEdjo8d 1/R/yEnDlxzNNizYj5lNgLHgdFn+fKFKfXFyB/+Seiz4vArvqO8ANZaKOp7Uw0fBmGbFO6GwYTaIg at8SZymaTvGHNbx66XJtf6T8pErhxV7KpAnSHZJgjxxGfdf5do0MGOqcLdZtXTkXWA6bHkmeH1Kvm m0YPIV/O/rmV5Wu5RKjJGxxUVq7ufADpH1mG7YwRmsRX+UmI74SPnioF+gf3js/Gd/qm9oaSM1Dq+ UhhvtYRwevV5picUNifw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jFd3a-0001ys-Ll; Sat, 21 Mar 2020 12:16:50 +0000 Received: from szxga04-in.huawei.com ([45.249.212.190] helo=huawei.com) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jFd3Y-0001wr-1M for linux-arm-kernel@lists.infradead.org; Sat, 21 Mar 2020 12:16:49 +0000 Received: from DGGEMS410-HUB.china.huawei.com (unknown [172.30.72.58]) by Forcepoint Email with ESMTP id 45230B185611B672D09E; Sat, 21 Mar 2020 20:16:37 +0800 (CST) Received: from DESKTOP-KKJBAGG.china.huawei.com (10.173.220.25) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.487.0; Sat, 21 Mar 2020 20:16:30 +0800 From: Zhenyu Ye To: , , , , , , , Subject: [RFC PATCH v3 1/4] arm64: Add level-hinted TLB invalidation helper to tlbi_user Date: Sat, 21 Mar 2020 20:16:18 +0800 Message-ID: <20200321121621.1600-2-yezhenyu2@huawei.com> X-Mailer: git-send-email 2.22.0.windows.1 In-Reply-To: <20200321121621.1600-1-yezhenyu2@huawei.com> References: <20200321121621.1600-1-yezhenyu2@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.173.220.25] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200321_051648_257673_A898B5DD X-CRM114-Status: GOOD ( 11.11 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.190 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, yezhenyu2@huawei.com, linux-kernel@vger.kernel.org, xiexiangyou@huawei.com, zhangshaokun@hisilicon.com, linux-mm@kvack.org, arm@kernel.org, prime.zeng@hisilicon.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Add a level-hinted parameter to __tlbi_user, which only gets used if ARMv8.4-TTL gets detected. ARMv8.4-TTL provides the TTL field in tlbi instruction to indicate the level of translation table walk holding the leaf entry for the address that is being invalidated. This patch set the default level value to 0. Signed-off-by: Zhenyu Ye --- arch/arm64/include/asm/tlbflush.h | 42 ++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index a3f70778a325..d141c080e494 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -89,6 +89,36 @@ __tlbi(op, arg); \ } while(0) +#define __tlbi_user_level(op, addr, level) \ + do { \ + u64 arg = addr; \ + \ + if (!arm64_kernel_unmapped_at_el0()) \ + break; \ + \ + if (cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL) && \ + level) { \ + u64 ttl = level; \ + \ + switch (PAGE_SIZE) { \ + case SZ_4K: \ + ttl |= 1 << 2; \ + break; \ + case SZ_16K: \ + ttl |= 2 << 2; \ + break; \ + case SZ_64K: \ + ttl |= 3 << 2; \ + break; \ + } \ + \ + arg &= ~TLBI_TTL_MASK; \ + arg |= FIELD_PREP(TLBI_TTL_MASK, ttl); \ + } \ + \ + __tlbi(op, (arg) | USER_ASID_FLAG); \ + } while (0) + /* * TLB Invalidation * ================ @@ -190,8 +220,8 @@ static inline void flush_tlb_page_nosync(struct vm_area_struct *vma, unsigned long addr = __TLBI_VADDR(uaddr, ASID(vma->vm_mm)); dsb(ishst); - __tlbi(vale1is, addr); - __tlbi_user(vale1is, addr); + __tlbi_level(vale1is, addr, 0); + __tlbi_user_level(vale1is, addr, 0); } static inline void flush_tlb_page(struct vm_area_struct *vma, @@ -231,11 +261,11 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, dsb(ishst); for (addr = start; addr < end; addr += stride) { if (last_level) { - __tlbi(vale1is, addr); - __tlbi_user(vale1is, addr); + __tlbi_level(vale1is, addr, 0); + __tlbi_user_level(vale1is, addr, 0); } else { - __tlbi(vae1is, addr); - __tlbi_user(vae1is, addr); + __tlbi_level(vae1is, addr, 0); + __tlbi_user_level(vae1is, addr, 0); } } dsb(ish); From patchwork Sat Mar 21 12:16:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhenyu Ye X-Patchwork-Id: 11451149 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A3645913 for ; Sat, 21 Mar 2020 12:17:36 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7FB3620757 for ; Sat, 21 Mar 2020 12:17:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="QD0b3dcF" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7FB3620757 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=moz5B4ZMiN2J8EXx7ZAd6auMHuKDAIYzJLORKkM57+M=; b=QD0b3dcFFBNj72 tWgOMnAX2bYFfkxLV7Q4F3JCObmEtHiGSaFXMB6e3H6/9MAunYS5G4QUsIhVQxKwV2j987Tf+qR9N ypOnNhzzmfXXA+2pT3hY7ZsylaNoQCbS7fWa0LGw7PyTGNgxgnkhqnjie+wXBaz/Y9K1+MvmEUSUX prhK+dRLSH5+jPf9GnsXn/IU7AhwlZcpdKMMq+SI5fj+rDUqFcbfDBLa3/VhxULqHaVcy2m2lj9ry euTnEp2VAPokLTCEXkpr/5CyLLk9GF6eDph7ymTwIGm6Lp4YEECC99Ppx4Ll/rHH6SoMvwNdeZHaf oOgMtDnBTVmbyY5UmkTg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jFd4H-0002df-Nc; Sat, 21 Mar 2020 12:17:33 +0000 Received: from szxga06-in.huawei.com ([45.249.212.32] helo=huawei.com) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jFd3d-0001wt-Hm for linux-arm-kernel@lists.infradead.org; Sat, 21 Mar 2020 12:16:55 +0000 Received: from DGGEMS410-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 4F697CC732300DE34742; Sat, 21 Mar 2020 20:16:42 +0800 (CST) Received: from DESKTOP-KKJBAGG.china.huawei.com (10.173.220.25) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.487.0; Sat, 21 Mar 2020 20:16:31 +0800 From: Zhenyu Ye To: , , , , , , , Subject: [RFC PATCH v3 2/4] mm: Add page table level flags to vm_flags Date: Sat, 21 Mar 2020 20:16:19 +0800 Message-ID: <20200321121621.1600-3-yezhenyu2@huawei.com> X-Mailer: git-send-email 2.22.0.windows.1 In-Reply-To: <20200321121621.1600-1-yezhenyu2@huawei.com> References: <20200321121621.1600-1-yezhenyu2@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.173.220.25] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200321_051653_795781_86E2335D X-CRM114-Status: GOOD ( 10.73 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.32 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, yezhenyu2@huawei.com, linux-kernel@vger.kernel.org, xiexiangyou@huawei.com, zhangshaokun@hisilicon.com, linux-mm@kvack.org, arm@kernel.org, prime.zeng@hisilicon.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Add VM_LEVEL_[PUD|PMD|PTE] to vm_flags to indicate which level of the page tables the vma is in. Those flags can be used to reduce the cost of TLB invalidation. These should be common flags for all architectures, however, those flags are only available in 64-bits system currently, because the lower-order flags are fully used. These flags are only used by ARM64 architecture now. See in next patch. Signed-off-by: Zhenyu Ye --- include/linux/mm.h | 10 ++++++++++ include/trace/events/mmflags.h | 15 ++++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 52269e56c514..63a04e02a221 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -313,6 +313,16 @@ extern unsigned int kobjsize(const void *objp); #endif #endif /* CONFIG_ARCH_HAS_PKEYS */ +#ifdef CONFIG_64BIT +# define VM_LEVEL_PUD BIT(37) /* vma is in pud-level of page table */ +# define VM_LEVEL_PMD BIT(38) /* vma is in pmd-level of page table */ +# define VM_LEVEL_PTE BIT(39) /* vma is in pte-level of page table */ +#else +# define VM_LEVEL_PUD 0 +# define VM_LEVEL_PMD 0 +# define VM_LEVEL_PTE 0 +#endif /* CONFIG_64BIT */ + #if defined(CONFIG_X86) # define VM_PAT VM_ARCH_1 /* PAT reserves whole VMA at once (x86) */ #elif defined(CONFIG_PPC) diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index a1675d43777e..9f13cfa96f9f 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -130,6 +130,16 @@ IF_HAVE_PG_IDLE(PG_idle, "idle" ) #define IF_HAVE_VM_SOFTDIRTY(flag,name) #endif +#ifdef CONFIG_64BIT +#define IF_HAVE_VM_LEVEL_PUD(flag,name) {flag, name} +#define IF_HAVE_VM_LEVEL_PMD(flag,name) {flag, name} +#define IF_HAVE_VM_LEVEL_PTE(flag,name) {flag, name} +#else +#define IF_HAVE_VM_LEVEL_PUD(flag,name) +#define IF_HAVE_VM_LEVEL_PMD(flag,name) +#define IF_HAVE_VM_LEVEL_PTE(flag,name) +#endif + #define __def_vmaflag_names \ {VM_READ, "read" }, \ {VM_WRITE, "write" }, \ @@ -161,7 +171,10 @@ IF_HAVE_VM_SOFTDIRTY(VM_SOFTDIRTY, "softdirty" ) \ {VM_MIXEDMAP, "mixedmap" }, \ {VM_HUGEPAGE, "hugepage" }, \ {VM_NOHUGEPAGE, "nohugepage" }, \ - {VM_MERGEABLE, "mergeable" } \ + {VM_MERGEABLE, "mergeable" }, \ +IF_HAVE_VM_LEVEL_PUD(VM_LEVEL_PUD, "pud-level" ), \ +IF_HAVE_VM_LEVEL_PMD(VM_LEVEL_PMD, "pmd-level" ), \ +IF_HAVE_VM_LEVEL_PTE(VM_LEVEL_PTE, "pte-level" ) \ #define show_vma_flags(flags) \ (flags) ? __print_flags(flags, "|", \ From patchwork Sat Mar 21 12:16:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhenyu Ye X-Patchwork-Id: 11451147 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6BBB5913 for ; Sat, 21 Mar 2020 12:17:22 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4496820722 for ; Sat, 21 Mar 2020 12:17:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="t7bU+9NH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4496820722 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NPLSbyaCAFI1g2mMclY1Zgr2J19bz3R63tmfTQ5vXZE=; b=t7bU+9NHEDEkGt yjqAWjLnVaxFCqceDXxTO/KCIRTYadQyVhiAxGMZqEmfCZm/VN+7Usw2Mzy2nmMbJMsU2HnQohPKV MS93lMWLiJgV8qqJhq+NyOu0wKCRjD+eS70kGh3F5QvaZ3r80MK1J5Xe5Jmqer2i4JLVjfPeOJkVq kxSgDmDErRM7UNfyn3g/1qNMXvHVjo9/W2/37SCQaDfx67eeebcceUbgJbs05i06pLo7gboIT28/B V/4KSyTmgT9TCyC2iEqQ82r8vS6eg73uRuNs1M4aJ/r7TbZLhy0ZNQdAiIwY4FD/14ujbFVQ0crJD IZNqxa/kfWkev2u1YClw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jFd42-0002PV-SS; Sat, 21 Mar 2020 12:17:18 +0000 Received: from szxga06-in.huawei.com ([45.249.212.32] helo=huawei.com) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jFd3a-0001x2-85 for linux-arm-kernel@lists.infradead.org; Sat, 21 Mar 2020 12:16:52 +0000 Received: from DGGEMS410-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 5892BC50B8209EFE8990; Sat, 21 Mar 2020 20:16:42 +0800 (CST) Received: from DESKTOP-KKJBAGG.china.huawei.com (10.173.220.25) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.487.0; Sat, 21 Mar 2020 20:16:32 +0800 From: Zhenyu Ye To: , , , , , , , Subject: [RFC PATCH v3 3/4] arm64: tlb: Use translation level hint in vm_flags Date: Sat, 21 Mar 2020 20:16:20 +0800 Message-ID: <20200321121621.1600-4-yezhenyu2@huawei.com> X-Mailer: git-send-email 2.22.0.windows.1 In-Reply-To: <20200321121621.1600-1-yezhenyu2@huawei.com> References: <20200321121621.1600-1-yezhenyu2@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.173.220.25] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200321_051650_470815_7FC58952 X-CRM114-Status: GOOD ( 10.40 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.32 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, yezhenyu2@huawei.com, linux-kernel@vger.kernel.org, xiexiangyou@huawei.com, zhangshaokun@hisilicon.com, linux-mm@kvack.org, arm@kernel.org, prime.zeng@hisilicon.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org This patch used the VM_LEVEL flags in vma->vm_flags to set the TTL field in tlbi instruction. Signed-off-by: Zhenyu Ye --- arch/arm64/include/asm/mmu.h | 2 ++ arch/arm64/include/asm/tlbflush.h | 14 ++++++++------ arch/arm64/mm/mmu.c | 14 ++++++++++++++ 3 files changed, 24 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index e4d862420bb4..b3f410fee512 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -88,6 +88,8 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot); extern void mark_linear_text_alias_ro(void); extern bool kaslr_requires_kpti(void); +extern unsigned int get_vma_level(struct vm_area_struct *vma); + #define INIT_MM_CONTEXT(name) \ .pgd = init_pg_dir, diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index d141c080e494..93bb09fdfafd 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -218,10 +218,11 @@ static inline void flush_tlb_page_nosync(struct vm_area_struct *vma, unsigned long uaddr) { unsigned long addr = __TLBI_VADDR(uaddr, ASID(vma->vm_mm)); + unsigned int level = get_vma_level(vma); dsb(ishst); - __tlbi_level(vale1is, addr, 0); - __tlbi_user_level(vale1is, addr, 0); + __tlbi_level(vale1is, addr, level); + __tlbi_user_level(vale1is, addr, level); } static inline void flush_tlb_page(struct vm_area_struct *vma, @@ -242,6 +243,7 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long stride, bool last_level) { unsigned long asid = ASID(vma->vm_mm); + unsigned int level = get_vma_level(vma); unsigned long addr; start = round_down(start, stride); @@ -261,11 +263,11 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, dsb(ishst); for (addr = start; addr < end; addr += stride) { if (last_level) { - __tlbi_level(vale1is, addr, 0); - __tlbi_user_level(vale1is, addr, 0); + __tlbi_level(vale1is, addr, level); + __tlbi_user_level(vale1is, addr, level); } else { - __tlbi_level(vae1is, addr, 0); - __tlbi_user_level(vae1is, addr, 0); + __tlbi_level(vae1is, addr, level); + __tlbi_user_level(vae1is, addr, level); } } dsb(ish); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 128f70852bf3..e6a1221cd86b 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -60,6 +60,20 @@ static pud_t bm_pud[PTRS_PER_PUD] __page_aligned_bss __maybe_unused; static DEFINE_SPINLOCK(swapper_pgdir_lock); +inline unsigned int get_vma_level(struct vm_area_struct *vma) +{ + unsigned int level = 0; + if (vma->vm_flags & VM_LEVEL_PUD) + level = 1; + else if (vma->vm_flags & VM_LEVEL_PMD) + level = 2; + else if (vma->vm_flags & VM_LEVEL_PTE) + level = 3; + + vma->vm_flags &= ~(VM_LEVEL_PUD | VM_LEVEL_PMD | VM_LEVEL_PTE); + return level; +} + void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) { pgd_t *fixmap_pgdp; From patchwork Sat Mar 21 12:16:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhenyu Ye X-Patchwork-Id: 11451145 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2EC1D13B1 for ; Sat, 21 Mar 2020 12:17:02 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0B2C520722 for ; Sat, 21 Mar 2020 12:17:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="KIYGt4S7" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B2C520722 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JCZgEQFzQLgxc6MteqzhnhAWcQbVbEniZAuLOKdKCtA=; b=KIYGt4S70heytF rEwZc6mBPKRznv+rEJtdnekMQCUz1nlUjmKfzNYzE7o9Mr+ZkRVbJq1vcP7jWtZhjMF3mbAuvBCOm J9Fs5j7BZRjauBjBx58KFZgZMFRU4WQCMJ+vivvAk4br34XhcNjde0xIxUUMYOvgVudJ+npya+kEo 1sDjkw+CCHw4AQ3QnqwuStODJFeBVjZCo9WEAsUre2FZbr/tkqhQyvwAhgcgVeWvn0FbmfJsA7B4y hDwWfbGkconTHn8M3UNIJ9w6xX6UUrHjBel/9XJvPC7SraTSHjgJSzEmsHsKtj4Dfxr/yzHBBSuo2 bE63pPfKhXcuqOveSdvQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jFd3j-00028o-JM; Sat, 21 Mar 2020 12:16:59 +0000 Received: from szxga06-in.huawei.com ([45.249.212.32] helo=huawei.com) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jFd3Y-0001wu-8g for linux-arm-kernel@lists.infradead.org; Sat, 21 Mar 2020 12:16:49 +0000 Received: from DGGEMS410-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 60DF2373D7F53E2F57F7; Sat, 21 Mar 2020 20:16:42 +0800 (CST) Received: from DESKTOP-KKJBAGG.china.huawei.com (10.173.220.25) by DGGEMS410-HUB.china.huawei.com (10.3.19.210) with Microsoft SMTP Server id 14.3.487.0; Sat, 21 Mar 2020 20:16:33 +0800 From: Zhenyu Ye To: , , , , , , , Subject: [RFC PATCH v3 4/4] mm: Set VM_LEVEL flags in some tlb_flush functions Date: Sat, 21 Mar 2020 20:16:21 +0800 Message-ID: <20200321121621.1600-5-yezhenyu2@huawei.com> X-Mailer: git-send-email 2.22.0.windows.1 In-Reply-To: <20200321121621.1600-1-yezhenyu2@huawei.com> References: <20200321121621.1600-1-yezhenyu2@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.173.220.25] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200321_051648_489800_A08FEF74 X-CRM114-Status: GOOD ( 14.53 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.32 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, yezhenyu2@huawei.com, linux-kernel@vger.kernel.org, xiexiangyou@huawei.com, zhangshaokun@hisilicon.com, linux-mm@kvack.org, arm@kernel.org, prime.zeng@hisilicon.com, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The relevant functions are: tlb_flush in asm/tlb.h get_clear_flush and clear_flush in mm/hugetlbpage.c flush_pmd|pud_tlb_range in asm-generic/patable.h do_huge_pmd_numa_page and move_huge_pmd in mm/huge_memory.c Signed-off-by: Zhenyu Ye --- arch/arm64/include/asm/tlb.h | 12 ++++++++++++ arch/arm64/mm/hugetlbpage.c | 4 ++-- include/asm-generic/pgtable.h | 16 ++++++++++++++-- mm/huge_memory.c | 8 +++++++- 4 files changed, 35 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index b76df828e6b7..77fe942b30b6 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -27,6 +27,18 @@ static inline void tlb_flush(struct mmu_gather *tlb) bool last_level = !tlb->freed_tables; unsigned long stride = tlb_get_unmap_size(tlb); + /* + * mm_gather tracked which levels of the page tables + * have been cleared, we can use this info to set + * vm->vm_flags. + */ + if (tlb->cleared_ptes) + vma.vm_flags |= VM_LEVEL_PTE; + else if (tlb->cleared_pmds) + vma.vm_flags |= VM_LEVEL_PMD; + else if (tlb->cleared_puds) + vma.vm_flags |= VM_LEVEL_PUD; + /* * If we're tearing down the address space then we only care about * invalidating the walk-cache, since the ASID allocator won't diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index bbeb6a5a6ba6..c35a1bd06bd0 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -140,7 +140,7 @@ static pte_t get_clear_flush(struct mm_struct *mm, } if (valid) { - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, VM_LEVEL_PTE); flush_tlb_range(&vma, saddr, addr); } return orig_pte; @@ -161,7 +161,7 @@ static void clear_flush(struct mm_struct *mm, unsigned long pgsize, unsigned long ncontig) { - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, VM_LEVEL_PTE); unsigned long i, saddr = addr; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index e2e2bef07dd2..391e704faf7a 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -1160,8 +1160,20 @@ static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) * invalidate the entire TLB which is not desitable. * e.g. see arch/arc: flush_pmd_tlb_range */ -#define flush_pmd_tlb_range(vma, addr, end) flush_tlb_range(vma, addr, end) -#define flush_pud_tlb_range(vma, addr, end) flush_tlb_range(vma, addr, end) +#define flush_pmd_tlb_range(vma, addr, end) \ + do { \ + vma->vm_flags &= ~(VM_LEVEL_PUD | VM_LEVEL_PTE); \ + vma->vm_flags |= VM_LEVEL_PMD; \ + flush_tlb_range(vma, addr, end); \ + } while (0) + +#define flush_pud_tlb_range(vma, addr, end) \ + do { \ + vma->vm_flags &= ~(VM_LEVEL_PMD | VM_LEVEL_PTE); \ + vma->vm_flags |= VM_LEVEL_PUD; \ + flush_tlb_range(vma, addr, end); \ + } while (0) + #else #define flush_pmd_tlb_range(vma, addr, end) BUILD_BUG() #define flush_pud_tlb_range(vma, addr, end) BUILD_BUG() diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b08b199f9a11..f28ced8d298e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1646,6 +1646,8 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd) * mapping or not. Hence use the tlb range variant */ if (mm_tlb_flush_pending(vma->vm_mm)) { + vma->vm_flags &= ~(VM_LEVEL_PUD | VM_LEVEL_PTE); + vma->vm_flags |= VM_LEVEL_PMD; flush_tlb_range(vma, haddr, haddr + HPAGE_PMD_SIZE); /* * change_huge_pmd() released the pmd lock before @@ -1917,8 +1919,12 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, } pmd = move_soft_dirty_pmd(pmd); set_pmd_at(mm, new_addr, new_pmd, pmd); - if (force_flush) + if (force_flush) { + vma->vm_flags &= ~(VM_LEVEL_PUD | VM_LEVEL_PTE); + vma->vm_flags |= VM_LEVEL_PMD; flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE); + } + if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl);