From patchwork Mon Aug 14 10:38:52 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Murzin X-Patchwork-Id: 9898513 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id BB91C602BA for ; Mon, 14 Aug 2017 10:40:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AA1BB2074F for ; Mon, 14 Aug 2017 10:40:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D11322B1F; Mon, 14 Aug 2017 10:40:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 7280D2074F for ; Mon, 14 Aug 2017 10:40:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=vLSuZVqwXZpfE6KZ8vmNSi4JwjSlyReLjof2E0/GB3g=; b=ghqg+oPOmRdutPGoykVl654km1 CFXV1nNqKntegmKdKWnC8nqFeX3Xx8q18yxWmrluaiFjJ2CQh8jFhdTMr26pECbyfkazkUEw3lplq RHMt2BuvD5zIhKBahPzYW5geDiBVm7HSPCn6S8jxoLyWK7CTqw1SY/kFriSJw3ueu+0AN+JredvIz vcOXAUd+CO3WnCR3WuNsKjqRNJUYQs+F2OakY7tBbsvGEqb2dwem6ttjufsVVKbPe3AUnT5vSgxkR D7HuE7MSx2oQBtjHZ9QXyyhOTErn3kTzBa+9BN9ttCkmnFMWRnVieCAVSOMIuSiD2jjQx7An6bu3A /uUB25Ew==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dhCnG-0000Vn-DW; Mon, 14 Aug 2017 10:40:22 +0000 Received: from foss.arm.com ([217.140.101.70]) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dhCmV-0006vW-FL for linux-arm-kernel@lists.infradead.org; Mon, 14 Aug 2017 10:39:39 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5ED3015A2; Mon, 14 Aug 2017 03:39:16 -0700 (PDT) Received: from login2.euhpc.arm.com (login2.euhpc.arm.com [10.6.26.144]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 23ACB3F3E1; Mon, 14 Aug 2017 03:39:15 -0700 (PDT) From: Vladimir Murzin To: linux-arm-kernel@lists.infradead.org Subject: [RFC PATCH 1/3] arm64: mm: Support Common Not Private translations Date: Mon, 14 Aug 2017 11:38:52 +0100 Message-Id: <1502707134-10853-2-git-send-email-vladimir.murzin@arm.com> X-Mailer: git-send-email 2.0.0 In-Reply-To: <1502707134-10853-1-git-send-email-vladimir.murzin@arm.com> References: <1502707134-10853-1-git-send-email-vladimir.murzin@arm.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170814_033935_593606_F327EACC X-CRM114-Status: GOOD ( 20.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, catalin.marinas@arm.com, james.morse@arm.com, kbuild-all@01.org, marc.zyngier@arm.com MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Common Not Private (CNP) is a feature of ARMv8.2 extension which allows translation table entries to be shared between different PEs in the same inner shareable domain, so the hardware can use this fact to optimise the caching of such entries in the TLB. CNP occupies one bit in TTBRx_ELy and VTTBR_EL2, which advertises to the hardware that the translation table entries pointed to by this TTBR are the same as every PE in the same inner shareable domain for which the equivalent TTBR also has CNP bit set. In case CNP bit is set but TTBR does not point at the same translation table entries, then the system is mis-configured, so the results of translations are UNPREDICTABLE. This patch adds support for Common Not Private translations on different exceptions levels: (1) For EL0 there are a few cases we need to care of changes in TTBR0_EL1: - a switch to idmap - software emulated PAN in these cases we make sure that CNP is set for non-zero ASIDs only. (2) For EL1 we postpone setting CNP till all cpus are up and rely on cpufeature framework to 1) patch the code which is sensitive to CNP and 2) update TTBR1_EL1 with CNP bit set. The only case where TTBR1_EL1 can be reprogrammed is hibirnation, so the code there is changed to save raw TTBR1_EL1 and blindly restore it on resume. Signed-off-by: Vladimir Murzin --- arch/arm64/Kconfig | 12 ++++++++++++ arch/arm64/include/asm/asm-uaccess.h | 2 ++ arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/include/asm/cpufeature.h | 6 ++++++ arch/arm64/include/asm/memory.h | 1 + arch/arm64/include/asm/mmu_context.h | 14 ++++++++++++++ arch/arm64/include/asm/uaccess.h | 10 ++++++++-- arch/arm64/kernel/cpufeature.c | 22 ++++++++++++++++++++++ arch/arm64/kernel/hibernate.c | 2 +- arch/arm64/mm/proc.S | 5 +++++ 10 files changed, 73 insertions(+), 4 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index dfd9086..ce91a70 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -960,6 +960,18 @@ config ARM64_UAO regular load/store instructions if the cpu does not implement the feature. +config ARM64_CNP + bool "Enable support for Common Not Private (CNP) translations" + default y + help + Common Not Private (CNP) allows translation table entries to + be shared between different PEs in the same inner shareable + domain, so the hardware can use this fact to optimise the + caching of such entries in the TLB. + + Selecting this option allows the CNP feature to be detected + at runtime, and does not affect PEs that do not implement + this feature. endmenu config ARM64_MODULE_CMODEL_LARGE diff --git a/arch/arm64/include/asm/asm-uaccess.h b/arch/arm64/include/asm/asm-uaccess.h index ecd9788..a3e452e 100644 --- a/arch/arm64/include/asm/asm-uaccess.h +++ b/arch/arm64/include/asm/asm-uaccess.h @@ -12,6 +12,8 @@ #ifdef CONFIG_ARM64_SW_TTBR0_PAN .macro __uaccess_ttbr0_disable, tmp1 mrs \tmp1, ttbr1_el1 // swapper_pg_dir + bic \tmp1, \tmp1, #TTBR_CNP_BIT // unconditionally clear CNP bit to avoid alternatives + // inside alternatives add \tmp1, \tmp1, #SWAPPER_DIR_SIZE // reserved_ttbr0 at the end of swapper_pg_dir msr ttbr0_el1, \tmp1 // set reserved TTBR0_EL1 isb diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index 8d2272c..5ccb3f7 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -39,7 +39,8 @@ #define ARM64_WORKAROUND_QCOM_FALKOR_E1003 18 #define ARM64_WORKAROUND_858921 19 #define ARM64_WORKAROUND_CAVIUM_30115 20 +#define ARM64_HAS_CNP 21 -#define ARM64_NCAPS 21 +#define ARM64_NCAPS 22 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 428ee1f..f7c75ac 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -262,6 +262,12 @@ static inline bool system_uses_ttbr0_pan(void) !cpus_have_const_cap(ARM64_HAS_PAN); } +static inline bool system_supports_cnp(void) +{ + return IS_ENABLED(CONFIG_ARM64_CNP) && + cpus_have_const_cap(ARM64_HAS_CNP); +} + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h index ef39dcb..14bd7e6 100644 --- a/arch/arm64/include/asm/memory.h +++ b/arch/arm64/include/asm/memory.h @@ -77,6 +77,7 @@ #define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) #define FIXADDR_TOP (PCI_IO_START - SZ_2M) #define TASK_SIZE_64 (UL(1) << VA_BITS) +#define TTBR_CNP_BIT (UL(1) << 0) #ifdef CONFIG_COMPAT #define TASK_SIZE_32 UL(0x100000000) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 3257895a..c8adce2 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -135,6 +135,17 @@ static inline void cpu_replace_ttbr1(pgd_t *pgd) phys_addr_t pgd_phys = virt_to_phys(pgd); + if (system_supports_cnp()) { + /* + * cpu_replace_ttbr1() is used when there's a boot CPU up + * (i.e. cpufeture framework is not up yet) and latter only + * when we enable CNP via cpufeature's enable() callback. + */ + BUG_ON(pgd != swapper_pg_dir); + + pgd_phys |= TTBR_CNP_BIT; + } + replace_phys = (void *)__pa_symbol(idmap_cpu_replace_ttbr1); cpu_install_idmap(); @@ -178,6 +189,9 @@ static inline void update_saved_ttbr0(struct task_struct *tsk, BUG_ON(mm->pgd == swapper_pg_dir); task_thread_info(tsk)->ttbr0 = virt_to_phys(mm->pgd) | ASID(mm) << 48; + + if (system_supports_cnp() && ASID(mm)) + task_thread_info(tsk)->ttbr0 |= TTBR_CNP_BIT; } } #else diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h index fab46a0..f65a96c 100644 --- a/arch/arm64/include/asm/uaccess.h +++ b/arch/arm64/include/asm/uaccess.h @@ -104,8 +104,14 @@ static inline void __uaccess_ttbr0_disable(void) { unsigned long ttbr; - /* reserved_ttbr0 placed at the end of swapper_pg_dir */ - ttbr = read_sysreg(ttbr1_el1) + SWAPPER_DIR_SIZE; + /* + * reserved_ttbr0 is placed at the end of swapper_pg_dir. + * When CNP is in use, TTBR1 may have the CNP bit set, but the + * reserved_ttbr should only be used without CNP. + */ + ttbr = read_sysreg(ttbr1_el1); + ttbr &= ~TTBR_CNP_BIT; + ttbr += SWAPPER_DIR_SIZE; write_sysreg(ttbr, ttbr0_el1); isb(); } diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9f9e0064..6beeaa2 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -101,6 +101,7 @@ EXPORT_SYMBOL(cpu_hwcap_keys); static bool __maybe_unused cpufeature_pan_not_uao(const struct arm64_cpu_capabilities *entry, int __unused); +static int cpu_enable_cnp(void *__unused);; /* * NOTE: Any changes to the visibility of features should be kept in @@ -888,6 +889,19 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .min_field_value = 0, .matches = has_no_fpsimd, }, +#ifdef CONFIG_ARM64_CNP + { + .desc = "Common not Private translations", + .capability = ARM64_HAS_CNP, + .def_scope = SCOPE_SYSTEM, + .matches = has_cpuid_feature, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_CNP_SHIFT, + .min_field_value = 1, + .enable = cpu_enable_cnp, + }, +#endif {}, }; @@ -1198,6 +1212,14 @@ cpufeature_pan_not_uao(const struct arm64_cpu_capabilities *entry, int __unused) return (cpus_have_const_cap(ARM64_HAS_PAN) && !cpus_have_const_cap(ARM64_HAS_UAO)); } +#ifdef CONFIG_ARM64_CNP +static int cpu_enable_cnp(void *__unused) +{ + cpu_replace_ttbr1(swapper_pg_dir); + return 0; +} +#endif /* CONFIG_ARM64_CNP */ + /* * We emulate only the following system register space. * Op0 = 0x3, CRn = 0x0, Op1 = 0x0, CRm = [0, 4 - 7] diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index a44e139..e864e3f 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -124,7 +124,7 @@ int arch_hibernation_header_save(void *addr, unsigned int max_size) return -EOVERFLOW; arch_hdr_invariants(&hdr->invariants); - hdr->ttbr1_el1 = __pa_symbol(swapper_pg_dir); + hdr->ttbr1_el1 = read_sysreg(ttbr1_el1); hdr->reenter_kernel = _cpu_resume; /* We can't use __hyp_get_vectors() because kvm may still be loaded */ diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 877d42f..1c94030 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -141,6 +141,11 @@ ENTRY(cpu_do_switch_mm) pre_ttbr0_update_workaround x0, x2, x3 mmid x1, x1 // get mm->context.id bfi x0, x1, #48, #16 // set the ASID +alternative_if ARM64_HAS_CNP + cbz x1, 1f + orr x0, x0, #TTBR_CNP_BIT +1: +alternative_else_nop_endif msr ttbr0_el1, x0 // set TTBR0 isb post_ttbr0_update_workaround