From patchwork Wed Dec 7 20:04:31 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christopher Covington X-Patchwork-Id: 9465147 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C606260236 for ; Wed, 7 Dec 2016 20:06:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A6E38283BC for ; Wed, 7 Dec 2016 20:06:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9B91E28567; Wed, 7 Dec 2016 20:06:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.9]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 10690283BC for ; Wed, 7 Dec 2016 20:06:49 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.85_2 #1 (Red Hat Linux)) id 1cEiSr-0000TT-D5; Wed, 07 Dec 2016 20:05:17 +0000 Received: from smtp.codeaurora.org ([198.145.29.96]) by bombadil.infradead.org with esmtps (Exim 4.85_2 #1 (Red Hat Linux)) id 1cEiSl-0008BS-IO for linux-arm-kernel@lists.infradead.org; Wed, 07 Dec 2016 20:05:12 +0000 Received: by smtp.codeaurora.org (Postfix, from userid 1000) id 029B7614F9; Wed, 7 Dec 2016 20:04:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1481141091; bh=1MCOoU7pGL44HQwLu4gzNz52ZaRcbNfu5ynmMNrmtSc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jZ9+ewDGnoj5eratDDYcYsFSUS9qLdHaTWvyo/U3WDr1QKrghSnlivaqo8C3Hp32M Fnk8AamAaUO6d0NXZZ8/nf8A5r6l1yre4Yq8zL4o2E26sC2ZEUnjMYpzPjoHb9wips wTzD6BmB9LtLVTASNBBu4LH++T9BPdafp0i0FyfU= Received: from illium.qualcomm.com (global_nat1_iad_fw.qualcomm.com [129.46.232.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: cov@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id B633F6120C; Wed, 7 Dec 2016 20:04:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1481141089; bh=1MCOoU7pGL44HQwLu4gzNz52ZaRcbNfu5ynmMNrmtSc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l57cpGs+4Zq/drm5I0PqKgxhgPsNp/rZ+Po3LTvy9fOMXJ5W09c63E0IUf9PV9Qid 58AaUt0NHln7HVM+rv9VRCxHHgCzOjkhfcp1+uGN0+ikhT43yMcREx6Ud1e0zBQT0R t/8SxqE6P9RmNR2z3rTFK/UK7NJHEX2oXBV8opE8= DMARC-Filter: OpenDMARC Filter v1.3.1 smtp.codeaurora.org B633F6120C Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=pass smtp.mailfrom=cov@codeaurora.org From: Christopher Covington To: Catalin Marinas , Will Deacon , Paolo Bonzini , =?UTF-8?q?Radim=20Kr=C4=8Dm=C3=A1=C5=99?= , Christoffer Dall , Marc Zyngier , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu Subject: [PATCH] arm64: Work around Falkor erratum 1009 Date: Wed, 7 Dec 2016 15:04:31 -0500 Message-Id: <20161207200431.4587-1-cov@codeaurora.org> X-Mailer: git-send-email 2.9.3 In-Reply-To: <20161207200028.4420-1-cov@codeaurora.org> References: <20161207200028.4420-1-cov@codeaurora.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20161207_120511_824597_D6C9F7C5 X-CRM114-Status: GOOD ( 13.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Christopher Covington , Shanker Donthineni MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Shanker Donthineni During a TLB invalidate sequence targeting the inner shareable domain, Falkor may prematurely complete the DSB before all loads and stores using the old translation are observed; instruction fetches are not subject to the conditions of this erratum. Signed-off-by: Shanker Donthineni Signed-off-by: Christopher Covington --- arch/arm64/Kconfig | 10 +++++++++ arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/include/asm/tlbflush.h | 43 +++++++++++++++++++++++++++++++++++++++ arch/arm64/kernel/cpu_errata.c | 7 +++++++ arch/arm64/kvm/hyp/tlb.c | 39 ++++++++++++++++++++++++++++++----- 5 files changed, 96 insertions(+), 6 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1004a3d..125440f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -485,6 +485,16 @@ config QCOM_FALKOR_ERRATUM_E1003 If unsure, say Y. +config QCOM_FALKOR_ERRATUM_E1009 + bool "Falkor E1009: Prematurely complete a DSB after a TLBI" + default y + help + Falkor CPU may prematurely complete a DSB following a TLBI xxIS + invalidate maintenance operations. Repeat the TLBI operation one + more time to fix the issue. + + If unsure, say Y. + endmenu diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index cb6a8c2..5357d7f 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -35,7 +35,8 @@ #define ARM64_HYP_OFFSET_LOW 14 #define ARM64_MISMATCHED_CACHE_LINE_SIZE 15 #define ARM64_WORKAROUND_QCOM_FALKOR_E1003 16 +#define ARM64_WORKAROUND_QCOM_FALKOR_E1009 17 -#define ARM64_NCAPS 17 +#define ARM64_NCAPS 18 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index deab523..03bafc5 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -23,6 +23,7 @@ #include #include +#include /* * Raw TLBI operations. @@ -94,6 +95,13 @@ static inline void flush_tlb_all(void) dsb(ishst); __tlbi(vmalle1is); dsb(ish); + asm volatile(ALTERNATIVE( + "nop \n" + "nop \n", + "tlbi vmalle1is \n" + "dsb ish \n", + ARM64_WORKAROUND_QCOM_FALKOR_E1009) + : :); isb(); } @@ -104,6 +112,13 @@ static inline void flush_tlb_mm(struct mm_struct *mm) dsb(ishst); __tlbi(aside1is, asid); dsb(ish); + asm volatile(ALTERNATIVE( + "nop \n" + "nop \n", + "tlbi aside1is, %0 \n" + "dsb ish \n", + ARM64_WORKAROUND_QCOM_FALKOR_E1009) + : : "r" (asid)); } static inline void flush_tlb_page(struct vm_area_struct *vma, @@ -114,6 +129,13 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, dsb(ishst); __tlbi(vale1is, addr); dsb(ish); + asm volatile(ALTERNATIVE( + "nop \n" + "nop \n", + "tlbi vale1is, %0 \n" + "dsb ish \n", + ARM64_WORKAROUND_QCOM_FALKOR_E1009) + : : "r" (addr)); } /* @@ -145,6 +167,13 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, __tlbi(vae1is, addr); } dsb(ish); + asm volatile(ALTERNATIVE( + "nop \n" + "nop \n", + "tlbi vae1is, %0 \n" + "dsb ish \n", + ARM64_WORKAROUND_QCOM_FALKOR_E1009) + : : "r" (end)); } static inline void flush_tlb_range(struct vm_area_struct *vma, @@ -169,6 +198,13 @@ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12)) __tlbi(vaae1is, addr); dsb(ish); + asm volatile(ALTERNATIVE( + "nop \n" + "nop \n", + "tlbi vaae1is, %0 \n" + "dsb ish \n", + ARM64_WORKAROUND_QCOM_FALKOR_E1009) + : : "r" (end)); isb(); } @@ -183,6 +219,13 @@ static inline void __flush_tlb_pgtable(struct mm_struct *mm, __tlbi(vae1is, addr); dsb(ish); + asm volatile(ALTERNATIVE( + "nop \n" + "nop \n", + "tlbi vae1is, %0 \n" + "dsb ish \n", + ARM64_WORKAROUND_QCOM_FALKOR_E1009) + : : "r" (addr)); } #endif diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 3789e2f..8013579 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -137,6 +137,13 @@ const struct arm64_cpu_capabilities arm64_errata[] = { MIDR_RANGE(MIDR_QCOM_FALKOR_V1, 0x00, 0x00), }, #endif +#ifdef CONFIG_QCOM_FALKOR_ERRATUM_E1009 + { + .desc = "Qualcomm Falkor erratum E1009", + .capability = ARM64_WORKAROUND_QCOM_FALKOR_E1009, + MIDR_RANGE(MIDR_QCOM_FALKOR_V1, 0x00, 0x00), + }, +#endif { } }; diff --git a/arch/arm64/kvm/hyp/tlb.c b/arch/arm64/kvm/hyp/tlb.c index 88e2f2b..dfd3a77 100644 --- a/arch/arm64/kvm/hyp/tlb.c +++ b/arch/arm64/kvm/hyp/tlb.c @@ -16,6 +16,7 @@ */ #include +#include void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) { @@ -32,7 +33,14 @@ void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) * whole of Stage-1. Weep... */ ipa >>= 12; - asm volatile("tlbi ipas2e1is, %0" : : "r" (ipa)); + asm volatile("tlbi ipas2e1is, %0 \n" + ALTERNATIVE( + "nop \n" + "nop \n", + "dsb ish \n" + "tlbi ipas2e1is, %0 \n", + ARM64_WORKAROUND_QCOM_FALKOR_E1009) + : : "r" (ipa)); /* * We have to ensure completion of the invalidation at Stage-2, @@ -41,7 +49,14 @@ void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) * the Stage-1 invalidation happened first. */ dsb(ish); - asm volatile("tlbi vmalle1is" : : ); + asm volatile("tlbi vmalle1is \n" + ALTERNATIVE( + "nop \n" + "nop \n", + "dsb ish \n" + "tlbi vmalle1is \n", + ARM64_WORKAROUND_QCOM_FALKOR_E1009) + : : ); dsb(ish); isb(); @@ -57,7 +72,14 @@ void __hyp_text __kvm_tlb_flush_vmid(struct kvm *kvm) write_sysreg(kvm->arch.vttbr, vttbr_el2); isb(); - asm volatile("tlbi vmalls12e1is" : : ); + asm volatile("tlbi vmalls12e1is \n" + ALTERNATIVE( + "nop \n" + "nop \n", + "dsb ish \n" + "tlbi vmalls12e1is \n", + ARM64_WORKAROUND_QCOM_FALKOR_E1009) + : : ); dsb(ish); isb(); @@ -82,7 +104,14 @@ void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu) void __hyp_text __kvm_flush_vm_context(void) { dsb(ishst); - asm volatile("tlbi alle1is \n" - "ic ialluis ": : ); + asm volatile("tlbi alle1is \n" + ALTERNATIVE( + "nop \n" + "nop \n", + "dsb ish \n" + "tlbi alle1is \n", + ARM64_WORKAROUND_QCOM_FALKOR_E1009) + "ic ialluis \n" + : : ); dsb(ish); }