From patchwork Wed Aug 9 12:05:23 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dave Martin X-Patchwork-Id: 9890479 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id DC36E60363 for ; Wed, 9 Aug 2017 12:09:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C759728426 for ; Wed, 9 Aug 2017 12:09:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BC30028544; Wed, 9 Aug 2017 12:09:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5434228426 for ; Wed, 9 Aug 2017 12:09:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=tcC34T2jzsUA02EfO1YFmKtffZpW1t/hHjPUaIVoIm0=; b=ASfBHnXRtRDLdoevu5lFRHbBG7 fOBPyPMLRcLmH0VYQf7lGdJfstQO8nC/e7e2W6DSsaJK4Pp/fWDsVFV6Zi1UrnPINLbpW+yUMnBbF rL/zw2lucZear5c/9Bb8hG2kW+EQ9cqSsMvOGqy1vnkm3Gs7eGrugeAqE5fmVOx8HP9aGRkVM+JPd /hwRLZuC556DXZroM84gi6jU/uG5yf2ycy4a4x2QZv9KzDGV4e6pN6dFadCMog/6qbaTeyHUhmtDb 3kuuMbH2yBr2mGh0lVGodP3H5agBn5fzOdg0zUTWuVhlEuIAMw31AB+R8avxx6ZD8aFbCUYK5CUU+ RP4IXuhQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dfPmv-0006h9-RQ; Wed, 09 Aug 2017 12:08:38 +0000 Received: from merlin.infradead.org ([205.233.59.134]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1dfPm7-00067m-29 for linux-arm-kernel@bombadil.infradead.org; Wed, 09 Aug 2017 12:07:47 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=s2QK+KHgB7eZqMAm9lnD+EpxhVjLRbVzx6edJpettpE=; b=rNjpFqF9Zn2+rlWoIvqVywpxJ 2voQ0pxdlrGYOEgH0NwkrfKqV6hsu59ITGmeFKVOrh7L05er3WqD4aSZ2RtC5h84oREYc3TIzz3bt FfH01tB7+F9XCv5wA3Gs0RFbSi6yNYwnL7AWoPesUIRjf+LvD8sQ0HYgojXVLqmbLTb7e/2N2XKUE K49glVJPOSRzd9IUH9ehSKEeAZQXmLKJo629ds0I9lmupTm0x9STmxRXkzmcilfbECs89glgx6qJA ub6otTZPSZZs+dBI7Jf8OEXajtd2PypbQHF+kthLNUes2uqC1oVhnM5JPOUpSTg8jlx9ZEvzRBVgi iRCxgXpTw==; Received: from usa-sjc-mx-foss1.foss.arm.com ([217.140.101.70] helo=foss.arm.com) by merlin.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dfPm5-0004RE-Ce for linux-arm-kernel@lists.infradead.org; Wed, 09 Aug 2017 12:07:46 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 44A521596; Wed, 9 Aug 2017 05:07:45 -0700 (PDT) Received: from e103592.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id BEFB23F540; Wed, 9 Aug 2017 05:07:43 -0700 (PDT) From: Dave Martin To: linux-arm-kernel@lists.infradead.org Subject: [PATCH 17/27] arm64/sve: Preserve SVE registers around EFI runtime service calls Date: Wed, 9 Aug 2017 13:05:23 +0100 Message-Id: <1502280338-23002-18-git-send-email-Dave.Martin@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1502280338-23002-1-git-send-email-Dave.Martin@arm.com> References: <1502280338-23002-1-git-send-email-Dave.Martin@arm.com> X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, libc-alpha@sourceware.org, Ard Biesheuvel , Szabolcs Nagy , Catalin Marinas , Will Deacon , Richard Sandiford , kvmarm@lists.cs.columbia.edu MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP The EFI runtime services ABI allows EFI to make free use of the FPSIMD registers during EFI runtime service calls, subject to the callee-save requirements of the AArch64 procedure call standard. However, the SVE architecture allows upper bits of the SVE vector registers to be zeroed as a side-effect of FPSIMD V-register writes. This means that the SVE vector registers must be saved in their entirety in order to avoid data loss: non-SVE-aware EFI implementations cannot restore them correctly. The non-IRQ case is already handled gracefully by kernel_neon_begin(). For the IRQ case, this patch allocates a suitable per-CPU stash buffer for the full SVE register state and uses it to preserve the affected registers around EFI calls. It is currently unclear how the EFI runtime services ABI will be clarified with respect to SVE, so it safest to assume that the predicate registers and FFR must be saved and restored too. No attempt is made to restore the restore the vector length after a call, for now. It is deemed rather insane for EFI to change it, and contemporary EFI implementations certainly won't. Signed-off-by: Dave Martin Reviewed-by: Ard Biesheuvel --- arch/arm64/kernel/fpsimd.c | 53 ++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 49 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index b7fb836..c727b47 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -120,12 +120,14 @@ int sve_max_vl = -1; /* Set of available vector lengths, as vq_to_bit(vq): */ static DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); static bool sve_vq_map_finalised; +static void __percpu *efi_sve_state; #else /* ! CONFIG_ARM64_SVE */ /* Dummy declaration for code that will be optimised out: */ extern DECLARE_BITMAP(sve_vq_map, SVE_VQ_MAX); extern bool sve_vq_map_finalised; +extern void __percpu *efi_sve_state; #endif /* ! CONFIG_ARM64_SVE */ @@ -416,6 +418,23 @@ int sve_verify_vq_map(void) return ret; } +static void __init sve_kernel_mode_neon_setup(void) +{ + if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON)) + return; + + /* + * alloc_percpu() warns and prints a backtrace if this goes wrong. + * This is evidence of a crippled system and we are returning void, + * so no attempt is made to handle this situation here. + */ + BUG_ON(!sve_vl_valid(sve_max_vl)); + efi_sve_state = __alloc_percpu( + SVE_SIG_REGS_SIZE(sve_vq_from_vl(sve_max_vl)), 16); + if (!efi_sve_state) + panic("Cannot allocate percpu memory for EFI SVE save/restore"); +} + void __init sve_setup(void) { u64 zcr; @@ -455,6 +474,8 @@ void __init sve_setup(void) sve_max_vl); pr_info("SVE: default vector length %u bytes per vector\n", sve_default_vl); + + sve_kernel_mode_neon_setup(); } void fpsimd_release_thread(struct task_struct *dead_task) @@ -797,6 +818,7 @@ EXPORT_SYMBOL(kernel_neon_end); DEFINE_PER_CPU(struct fpsimd_state, efi_fpsimd_state); DEFINE_PER_CPU(bool, efi_fpsimd_state_used); +DEFINE_PER_CPU(bool, efi_sve_state_used); /* * EFI runtime services support functions @@ -825,7 +847,20 @@ void __efi_fpsimd_begin(void) if (may_use_simd()) kernel_neon_begin(); else { - fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state)); + /* + * If !efi_sve_state, SVE can't be in use yet and doesn't need + * preserving: + */ + if (system_supports_sve() && likely(efi_sve_state)) { + char *sve_state = this_cpu_ptr(efi_sve_state); + + __this_cpu_write(efi_sve_state_used, true); + + sve_save_state(sve_state + sve_ffr_offset(sve_max_vl), + &this_cpu_ptr(&efi_fpsimd_state)->fpsr); + } else + fpsimd_save_state(this_cpu_ptr(&efi_fpsimd_state)); + __this_cpu_write(efi_fpsimd_state_used, true); } } @@ -838,10 +873,20 @@ void __efi_fpsimd_end(void) if (!system_supports_fpsimd()) return; - if (__this_cpu_xchg(efi_fpsimd_state_used, false)) - fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state)); - else + if (!__this_cpu_xchg(efi_fpsimd_state_used, false)) kernel_neon_end(); + else + if (system_supports_sve() && + likely(__this_cpu_read(efi_sve_state_used))) { + char const *sve_state = this_cpu_ptr(efi_sve_state); + + sve_load_state(sve_state + sve_ffr_offset(sve_max_vl), + &this_cpu_ptr(&efi_fpsimd_state)->fpsr, + sve_vq_from_vl(sve_get_vl()) - 1); + + __this_cpu_write(efi_sve_state_used, false); + } else + fpsimd_load_state(this_cpu_ptr(&efi_fpsimd_state)); } #endif /* CONFIG_KERNEL_MODE_NEON */