From patchwork Thu Oct 4 14:05:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626237 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4D43115E2 for ; Thu, 4 Oct 2018 14:07:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3F478290F5 for ; Thu, 4 Oct 2018 14:07:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 33D3C291B8; Thu, 4 Oct 2018 14:07:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D63EC290F5 for ; Thu, 4 Oct 2018 14:07:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727569AbeJDU7c (ORCPT ); Thu, 4 Oct 2018 16:59:32 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36267 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727354AbeJDU7c (ORCPT ); Thu, 4 Oct 2018 16:59:32 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GN-0002ic-TO; Thu, 04 Oct 2018 16:06:00 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 01/11] x86/entry: remove _TIF_ALLWORK_MASK Date: Thu, 4 Oct 2018 16:05:37 +0200 Message-Id: <20181004140547.13014-2-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP There is no user of _TIF_ALLWORK_MASK since commit 21d375b6b34ff ("x86/entry/64: Remove the SYSCALL64 fast path"). Remove unused define _TIF_ALLWORK_MASK. Signed-off-by: Sebastian Andrzej Siewior Reviewed-by: Borislav Petkov --- arch/x86/include/asm/thread_info.h | 8 -------- 1 file changed, 8 deletions(-) diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thread_info.h index 2ff2a30a264f4..cd6920674b905 100644 --- a/arch/x86/include/asm/thread_info.h +++ b/arch/x86/include/asm/thread_info.h @@ -136,14 +136,6 @@ struct thread_info { _TIF_SECCOMP | _TIF_SYSCALL_TRACEPOINT | \ _TIF_NOHZ) -/* work to do on any return to user space */ -#define _TIF_ALLWORK_MASK \ - (_TIF_SYSCALL_TRACE | _TIF_NOTIFY_RESUME | _TIF_SIGPENDING | \ - _TIF_NEED_RESCHED | _TIF_SINGLESTEP | _TIF_SYSCALL_EMU | \ - _TIF_SYSCALL_AUDIT | _TIF_USER_RETURN_NOTIFY | _TIF_UPROBE | \ - _TIF_PATCH_PENDING | _TIF_NOHZ | _TIF_SYSCALL_TRACEPOINT | \ - _TIF_FSCHECK) - /* flags to check in __switch_to() */ #define _TIF_WORK_CTXSW \ (_TIF_IO_BITMAP|_TIF_NOCPUID|_TIF_NOTSC|_TIF_BLOCKSTEP|_TIF_SSBD) From patchwork Thu Oct 4 14:05:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626233 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1B3911515 for ; Thu, 4 Oct 2018 14:07:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0A880291B8 for ; Thu, 4 Oct 2018 14:07:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F3032291BE; Thu, 4 Oct 2018 14:07:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A5AD291B8 for ; Thu, 4 Oct 2018 14:07:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727654AbeJDU7d (ORCPT ); Thu, 4 Oct 2018 16:59:33 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36270 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727380AbeJDU7c (ORCPT ); Thu, 4 Oct 2018 16:59:32 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GO-0002ic-No; Thu, 04 Oct 2018 16:06:00 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 02/11] x86/fpu: add (__)make_fpregs_active helpers Date: Thu, 4 Oct 2018 16:05:38 +0200 Message-Id: <20181004140547.13014-3-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Rik van Riel Add helper function that ensures the floating point registers for the current task are active. Use with preemption disabled. Signed-off-by: Rik van Riel Signed-off-by: Sebastian Andrzej Siewior --- arch/x86/include/asm/fpu/internal.h | 27 ++++++++++++++++++++++----- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index a38bf5a1e37ad..16c4077ffc945 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -514,6 +514,26 @@ static inline void fpregs_activate(struct fpu *fpu) trace_x86_fpu_regs_activated(fpu); } +/* + * Load the FPU state for the current task. Call with preemption disabled. + */ +static inline void __fpregs_load_activate(struct fpu *fpu, int cpu) +{ + if (!fpregs_state_valid(fpu, cpu)) + copy_kernel_to_fpregs(&fpu->state); + fpregs_activate(fpu); +} + +static inline void __fpregs_changes_begin(void) +{ + preempt_disable(); +} + +static inline void __fpregs_changes_end(void) +{ + preempt_enable(); +} + /* * FPU state switching for scheduling. * @@ -553,11 +573,8 @@ static inline void switch_fpu_finish(struct fpu *new_fpu, int cpu) bool preload = static_cpu_has(X86_FEATURE_FPU) && new_fpu->initialized; - if (preload) { - if (!fpregs_state_valid(new_fpu, cpu)) - copy_kernel_to_fpregs(&new_fpu->state); - fpregs_activate(new_fpu); - } + if (preload) + __fpregs_load_activate(new_fpu, cpu); } /* From patchwork Thu Oct 4 14:05:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626229 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 70BA615E2 for ; Thu, 4 Oct 2018 14:06:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 604D92916D for ; Thu, 4 Oct 2018 14:06:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 54585291BC; Thu, 4 Oct 2018 14:06:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D6D772916D for ; Thu, 4 Oct 2018 14:06:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727705AbeJDU7e (ORCPT ); Thu, 4 Oct 2018 16:59:34 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36274 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727356AbeJDU7d (ORCPT ); Thu, 4 Oct 2018 16:59:33 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GP-0002ic-KC; Thu, 04 Oct 2018 16:06:01 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 03/11] x86/fpu: make __raw_xsave_addr() use feature number instead of mask Date: Thu, 4 Oct 2018 16:05:39 +0200 Message-Id: <20181004140547.13014-4-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Most users of __raw_xsave_addr() use a feature number, shift it to a mask and then __raw_xsave_addr() shifts it back to the feature number. Make __raw_xsave_addr() use the feature number as argument. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Sebastian Andrzej Siewior --- arch/x86/kernel/fpu/xstate.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index 87a57b7642d36..38d0b5ea40425 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -811,10 +811,8 @@ void fpu__resume_cpu(void) * * Note: does not work for compacted buffers. */ -void *__raw_xsave_addr(struct xregs_state *xsave, int xstate_feature_mask) +void *__raw_xsave_addr(struct xregs_state *xsave, int feature_nr) { - int feature_nr = fls64(xstate_feature_mask) - 1; - if (!xfeature_enabled(feature_nr)) { WARN_ON_FPU(1); return NULL; @@ -842,6 +840,7 @@ void *__raw_xsave_addr(struct xregs_state *xsave, int xstate_feature_mask) */ void *get_xsave_addr(struct xregs_state *xsave, int xstate_feature) { + int feature_nr; /* * Do we even *have* xsave state? */ @@ -869,7 +868,8 @@ void *get_xsave_addr(struct xregs_state *xsave, int xstate_feature) if (!(xsave->header.xfeatures & xstate_feature)) return NULL; - return __raw_xsave_addr(xsave, xstate_feature); + feature_nr = fls64(xstate_feature) - 1; + return __raw_xsave_addr(xsave, feature_nr); } EXPORT_SYMBOL_GPL(get_xsave_addr); @@ -1018,7 +1018,7 @@ int copy_xstate_to_kernel(void *kbuf, struct xregs_state *xsave, unsigned int of * Copy only in-use xstates: */ if ((header.xfeatures >> i) & 1) { - void *src = __raw_xsave_addr(xsave, 1 << i); + void *src = __raw_xsave_addr(xsave, i); offset = xstate_offsets[i]; size = xstate_sizes[i]; @@ -1104,7 +1104,7 @@ int copy_xstate_to_user(void __user *ubuf, struct xregs_state *xsave, unsigned i * Copy only in-use xstates: */ if ((header.xfeatures >> i) & 1) { - void *src = __raw_xsave_addr(xsave, 1 << i); + void *src = __raw_xsave_addr(xsave, i); offset = xstate_offsets[i]; size = xstate_sizes[i]; @@ -1161,7 +1161,7 @@ int copy_kernel_to_xstate(struct xregs_state *xsave, const void *kbuf) u64 mask = ((u64)1 << i); if (hdr.xfeatures & mask) { - void *dst = __raw_xsave_addr(xsave, 1 << i); + void *dst = __raw_xsave_addr(xsave, i); offset = xstate_offsets[i]; size = xstate_sizes[i]; @@ -1215,7 +1215,7 @@ int copy_user_to_xstate(struct xregs_state *xsave, const void __user *ubuf) u64 mask = ((u64)1 << i); if (hdr.xfeatures & mask) { - void *dst = __raw_xsave_addr(xsave, 1 << i); + void *dst = __raw_xsave_addr(xsave, i); offset = xstate_offsets[i]; size = xstate_sizes[i]; From patchwork Thu Oct 4 14:05:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626231 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF6C815E2 for ; Thu, 4 Oct 2018 14:07:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BF6B2291B8 for ; Thu, 4 Oct 2018 14:07:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B2AF5291E7; Thu, 4 Oct 2018 14:07:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 141832916D for ; Thu, 4 Oct 2018 14:07:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728039AbeJDVA0 (ORCPT ); Thu, 4 Oct 2018 17:00:26 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36281 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727690AbeJDU7e (ORCPT ); Thu, 4 Oct 2018 16:59:34 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GQ-0002ic-IK; Thu, 04 Oct 2018 16:06:02 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 04/11] x86/fpu: eager switch PKRU state Date: Thu, 4 Oct 2018 16:05:40 +0200 Message-Id: <20181004140547.13014-5-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Rik van Riel While most of a task's FPU state is only needed in user space, the protection keys need to be in place immediately after a context switch. The reason is that any accesses to userspace memory while running in kernel mode also need to abide by the memory permissions specified in the protection keys. The "eager switch" is a preparation for loading the FPU state on return to userland. Instead of decoupling PKRU state from xstate I update PKRU within xstate on write operations by the kernel. Signed-off-by: Rik van Riel [bigeasy: save pkru to xstate, no cache] Signed-off-by: Sebastian Andrzej Siewior --- arch/x86/include/asm/fpu/internal.h | 20 +++++++++++++++---- arch/x86/include/asm/fpu/xstate.h | 2 ++ arch/x86/include/asm/pgtable.h | 6 +----- arch/x86/include/asm/pkeys.h | 2 +- arch/x86/kernel/fpu/core.c | 2 +- arch/x86/mm/pkeys.c | 31 ++++++++++++++++++++++------- include/linux/pkeys.h | 2 +- 7 files changed, 46 insertions(+), 19 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 16c4077ffc945..956d967ca824a 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -570,11 +570,23 @@ switch_fpu_prepare(struct fpu *old_fpu, int cpu) */ static inline void switch_fpu_finish(struct fpu *new_fpu, int cpu) { - bool preload = static_cpu_has(X86_FEATURE_FPU) && - new_fpu->initialized; + bool load_fpu; - if (preload) - __fpregs_load_activate(new_fpu, cpu); + load_fpu = static_cpu_has(X86_FEATURE_FPU) && new_fpu->initialized; + if (!load_fpu) + return; + + __fpregs_load_activate(new_fpu, cpu); + +#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS + if (static_cpu_has(X86_FEATURE_OSPKE)) { + struct pkru_state *pk; + + pk = __raw_xsave_addr(&new_fpu->state.xsave, XFEATURE_PKRU); + if (pk->pkru != __read_pkru()) + __write_pkru(pk->pkru); + } +#endif } /* diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index 48581988d78c7..82227e222ca35 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -5,6 +5,7 @@ #include #include #include +#include /* Bit 63 of XCR0 is reserved for future expansion */ #define XFEATURE_MASK_EXTEND (~(XFEATURE_MASK_FPSSE | (1ULL << 63))) @@ -47,6 +48,7 @@ extern void __init update_regset_xstate_info(unsigned int size, void fpu__xstate_clear_all_cpu_caps(void); void *get_xsave_addr(struct xregs_state *xsave, int xstate); +void *__raw_xsave_addr(struct xregs_state *xsave, int feature_nr); const void *get_xsave_field_ptr(int xstate_field); int using_compacted_format(void); int copy_xstate_to_kernel(void *kbuf, struct xregs_state *xsave, unsigned int offset, unsigned int size); diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 690c0307afed0..80be15ba656f6 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -132,11 +132,7 @@ static inline u32 read_pkru(void) return 0; } -static inline void write_pkru(u32 pkru) -{ - if (boot_cpu_has(X86_FEATURE_OSPKE)) - __write_pkru(pkru); -} +void write_pkru(u32 pkru); static inline int pte_young(pte_t pte) { diff --git a/arch/x86/include/asm/pkeys.h b/arch/x86/include/asm/pkeys.h index 19b137f1b3beb..b184f916319e5 100644 --- a/arch/x86/include/asm/pkeys.h +++ b/arch/x86/include/asm/pkeys.h @@ -119,7 +119,7 @@ extern int arch_set_user_pkey_access(struct task_struct *tsk, int pkey, unsigned long init_val); extern int __arch_set_user_pkey_access(struct task_struct *tsk, int pkey, unsigned long init_val); -extern void copy_init_pkru_to_fpregs(void); +extern void pkru_set_init_value(void); static inline int vma_pkey(struct vm_area_struct *vma) { diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 2ea85b32421a0..72cd2e2a07194 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -373,7 +373,7 @@ static inline void copy_init_fpstate_to_fpregs(void) copy_kernel_to_fregs(&init_fpstate.fsave); if (boot_cpu_has(X86_FEATURE_OSPKE)) - copy_init_pkru_to_fpregs(); + pkru_set_init_value(); } /* diff --git a/arch/x86/mm/pkeys.c b/arch/x86/mm/pkeys.c index 6e98e0a7c9231..4409ada551c5e 100644 --- a/arch/x86/mm/pkeys.c +++ b/arch/x86/mm/pkeys.c @@ -18,6 +18,7 @@ #include /* boot_cpu_has, ... */ #include /* vma_pkey() */ +#include int __execute_only_pkey(struct mm_struct *mm) { @@ -123,6 +124,24 @@ int __arch_override_mprotect_pkey(struct vm_area_struct *vma, int prot, int pkey return vma_pkey(vma); } +void write_pkru(u32 pkru) +{ + struct pkru_state *pk; + + if (!boot_cpu_has(X86_FEATURE_OSPKE)) + return; + + pk = __raw_xsave_addr(¤t->thread.fpu.state.xsave, XFEATURE_PKRU); + /* + * Update the PKRU value in cstate and then in the CPU. A context + * switch between those two operation would load the new value from the + * updated xstate and then we would write (the same value) to the CPU. + */ + pk->pkru = pkru; + __write_pkru(pkru); + +} + #define PKRU_AD_KEY(pkey) (PKRU_AD_BIT << ((pkey) * PKRU_BITS_PER_PKEY)) /* @@ -143,20 +162,18 @@ u32 init_pkru_value = PKRU_AD_KEY( 1) | PKRU_AD_KEY( 2) | PKRU_AD_KEY( 3) | * we know the FPU regstiers are safe for use and we can use PKRU * directly. */ -void copy_init_pkru_to_fpregs(void) +void pkru_set_init_value(void) { u32 init_pkru_value_snapshot = READ_ONCE(init_pkru_value); + /* * Any write to PKRU takes it out of the XSAVE 'init * state' which increases context switch cost. Avoid - * writing 0 when PKRU was already 0. + * writing then same value which is already written. */ - if (!init_pkru_value_snapshot && !read_pkru()) + if (init_pkru_value_snapshot == read_pkru()) return; - /* - * Override the PKRU state that came from 'init_fpstate' - * with the baseline from the process. - */ + write_pkru(init_pkru_value_snapshot); } diff --git a/include/linux/pkeys.h b/include/linux/pkeys.h index 2955ba9760489..9a9efecc1388f 100644 --- a/include/linux/pkeys.h +++ b/include/linux/pkeys.h @@ -44,7 +44,7 @@ static inline bool arch_pkeys_enabled(void) return false; } -static inline void copy_init_pkru_to_fpregs(void) +static inline void pkru_set_init_value(void) { } From patchwork Thu Oct 4 14:05:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626223 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D8EBD15E2 for ; Thu, 4 Oct 2018 14:06:46 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C8AFE291BC for ; Thu, 4 Oct 2018 14:06:46 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BCCCF291CD; Thu, 4 Oct 2018 14:06:46 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 501DE291BC for ; Thu, 4 Oct 2018 14:06:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727746AbeJDU7f (ORCPT ); Thu, 4 Oct 2018 16:59:35 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36296 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727380AbeJDU7f (ORCPT ); Thu, 4 Oct 2018 16:59:35 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GR-0002ic-8G; Thu, 04 Oct 2018 16:06:03 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 05/11] x86/fpu: set PKRU state for kernel threads Date: Thu, 4 Oct 2018 16:05:41 +0200 Message-Id: <20181004140547.13014-6-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The PKRU value is not set for kernel threads because they do not have the ->initialized value set. As a result the kernel thread has a random PKRU value set which it inherits from the previous task. It has been suggested by Paolo Bonzini to set it for kernel threads, too because it might be a fix. I *think* this is not required because the kernel threads don't copy data to/from userland and don't have access to any userspace mm in general. However there is this use_mm(). If we gain a mm by use_mm() we don't have a matching PKRU value because those are per thread. It has been suggested to use 0 as the PKRU value but this would bypass PKRU. Set the initial (default) PKRU value for kernel threads. Suggested-by: Paolo Bonzini Signed-off-by: Sebastian Andrzej Siewior --- arch/x86/include/asm/fpu/internal.h | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 956d967ca824a..4ecaf4d22954e 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -573,20 +574,23 @@ static inline void switch_fpu_finish(struct fpu *new_fpu, int cpu) bool load_fpu; load_fpu = static_cpu_has(X86_FEATURE_FPU) && new_fpu->initialized; - if (!load_fpu) - return; - - __fpregs_load_activate(new_fpu, cpu); - #ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS if (static_cpu_has(X86_FEATURE_OSPKE)) { struct pkru_state *pk; - pk = __raw_xsave_addr(&new_fpu->state.xsave, XFEATURE_PKRU); - if (pk->pkru != __read_pkru()) - __write_pkru(pk->pkru); + if (!load_fpu) { + pkru_set_init_value(); + } else { + pk = __raw_xsave_addr(&new_fpu->state.xsave, + XFEATURE_PKRU); + if (pk->pkru != __read_pkru()) + __write_pkru(pk->pkru); + } } #endif + if (!load_fpu) + return; + __fpregs_load_activate(new_fpu, cpu); } /* From patchwork Thu Oct 4 14:05:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626227 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A5A8A1515 for ; Thu, 4 Oct 2018 14:06:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 96AE32916D for ; Thu, 4 Oct 2018 14:06:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8B1F9291E2; Thu, 4 Oct 2018 14:06:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 29C172916D for ; Thu, 4 Oct 2018 14:06:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728009AbeJDVAN (ORCPT ); Thu, 4 Oct 2018 17:00:13 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36297 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727696AbeJDU7f (ORCPT ); Thu, 4 Oct 2018 16:59:35 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GR-0002ic-Sa; Thu, 04 Oct 2018 16:06:04 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 06/11] x86/pkeys: make init_pkru_value static Date: Thu, 4 Oct 2018 16:05:42 +0200 Message-Id: <20181004140547.13014-7-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The variable init_pkru_value isn't used outside of this file. Make init_pkru_value static. Signed-off-by: Sebastian Andrzej Siewior Acked-by: Dave Hansen --- arch/x86/mm/pkeys.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/mm/pkeys.c b/arch/x86/mm/pkeys.c index 4409ada551c5e..a150984171684 100644 --- a/arch/x86/mm/pkeys.c +++ b/arch/x86/mm/pkeys.c @@ -150,6 +150,7 @@ void write_pkru(u32 pkru) * in the process's lifetime will not accidentally get access * to data which is pkey-protected later on. */ +static u32 init_pkru_value = PKRU_AD_KEY( 1) | PKRU_AD_KEY( 2) | PKRU_AD_KEY( 3) | PKRU_AD_KEY( 4) | PKRU_AD_KEY( 5) | PKRU_AD_KEY( 6) | PKRU_AD_KEY( 7) | PKRU_AD_KEY( 8) | PKRU_AD_KEY( 9) | From patchwork Thu Oct 4 14:05:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626221 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0997B15E2 for ; Thu, 4 Oct 2018 14:06:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE47D2916D for ; Thu, 4 Oct 2018 14:06:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E2573291BC; Thu, 4 Oct 2018 14:06:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 86B7E2916D for ; Thu, 4 Oct 2018 14:06:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727786AbeJDU7g (ORCPT ); Thu, 4 Oct 2018 16:59:36 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36307 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727356AbeJDU7g (ORCPT ); Thu, 4 Oct 2018 16:59:36 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GS-0002ic-N1; Thu, 04 Oct 2018 16:06:05 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 07/11] x86/pkeys: Drop the preempt-disable section Date: Thu, 4 Oct 2018 16:05:43 +0200 Message-Id: <20181004140547.13014-8-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Rik van Riel The fpu->initialized flag should not be changed underneath us. This might be a fallout during the removal of the LazyFPU support. The FPU is marked initialized as soon as the state has been set to an initial value. It does not signal if the CPU's FPU registers are loaded. Signed-off-by: Rik van Riel Signed-off-by: Sebastian Andrzej Siewior --- arch/x86/mm/pkeys.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/arch/x86/mm/pkeys.c b/arch/x86/mm/pkeys.c index a150984171684..eeb03b0f0f514 100644 --- a/arch/x86/mm/pkeys.c +++ b/arch/x86/mm/pkeys.c @@ -40,17 +40,13 @@ int __execute_only_pkey(struct mm_struct *mm) * dance to set PKRU if we do not need to. Check it * first and assume that if the execute-only pkey is * write-disabled that we do not have to set it - * ourselves. We need preempt off so that nobody - * can make fpregs inactive. + * ourselves. */ - preempt_disable(); if (!need_to_set_mm_pkey && current->thread.fpu.initialized && !__pkru_allows_read(read_pkru(), execute_only_pkey)) { - preempt_enable(); return execute_only_pkey; } - preempt_enable(); /* * Set up PKRU so that it denies access for everything From patchwork Thu Oct 4 14:05:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626219 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 389531515 for ; Thu, 4 Oct 2018 14:06:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 294512916D for ; Thu, 4 Oct 2018 14:06:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1D6F6291BC; Thu, 4 Oct 2018 14:06:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95EB92916D for ; Thu, 4 Oct 2018 14:06:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727815AbeJDU7i (ORCPT ); Thu, 4 Oct 2018 16:59:38 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36314 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727380AbeJDU7h (ORCPT ); Thu, 4 Oct 2018 16:59:37 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GT-0002ic-Jv; Thu, 04 Oct 2018 16:06:05 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 08/11] x86/fpu: Always store the registers in copy_fpstate_to_sigframe() Date: Thu, 4 Oct 2018 16:05:44 +0200 Message-Id: <20181004140547.13014-9-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Rik van Riel copy_fpstate_to_sigframe() has two callers and both invoke the function only if fpu->initialized is set. So the check in the function for ->initialized makes no sense. It might be a relict from the lazy-FPU time: If the FPU registers were "loaded" then we could save them directly. Otherwise (the FPU registers are not up to date) they are saved in the FPU struct and it could be used for memcpy(). Since the registers always loaded at this point, save them and copy later. This code is extracted from an earlier version of the patchset while there still was lazy-FPU on x86. This is a preparation for loading the FPU registers on return to userland. Signed-off-by: Rik van Riel Signed-off-by: Sebastian Andrzej Siewior --- arch/x86/include/asm/fpu/internal.h | 45 ---------------------------- arch/x86/kernel/fpu/signal.c | 46 +++++++---------------------- 2 files changed, 11 insertions(+), 80 deletions(-) diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index 4ecaf4d22954e..df8816be3efdd 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -126,22 +126,6 @@ extern void fpstate_sanitize_xstate(struct fpu *fpu); _ASM_EXTABLE_HANDLE(1b, 2b, ex_handler_fprestore) \ : output : input) -static inline int copy_fregs_to_user(struct fregs_state __user *fx) -{ - return user_insn(fnsave %[fx]; fwait, [fx] "=m" (*fx), "m" (*fx)); -} - -static inline int copy_fxregs_to_user(struct fxregs_state __user *fx) -{ - if (IS_ENABLED(CONFIG_X86_32)) - return user_insn(fxsave %[fx], [fx] "=m" (*fx), "m" (*fx)); - else if (IS_ENABLED(CONFIG_AS_FXSAVEQ)) - return user_insn(fxsaveq %[fx], [fx] "=m" (*fx), "m" (*fx)); - - /* See comment in copy_fxregs_to_kernel() below. */ - return user_insn(rex64/fxsave (%[fx]), "=m" (*fx), [fx] "R" (fx)); -} - static inline void copy_kernel_to_fxregs(struct fxregs_state *fx) { if (IS_ENABLED(CONFIG_X86_32)) { @@ -352,35 +336,6 @@ static inline void copy_kernel_to_xregs(struct xregs_state *xstate, u64 mask) XSTATE_XRESTORE(xstate, lmask, hmask); } -/* - * Save xstate to user space xsave area. - * - * We don't use modified optimization because xrstor/xrstors might track - * a different application. - * - * We don't use compacted format xsave area for - * backward compatibility for old applications which don't understand - * compacted format of xsave area. - */ -static inline int copy_xregs_to_user(struct xregs_state __user *buf) -{ - int err; - - /* - * Clear the xsave header first, so that reserved fields are - * initialized to zero. - */ - err = __clear_user(&buf->header, sizeof(buf->header)); - if (unlikely(err)) - return -EFAULT; - - stac(); - XSTATE_OP(XSAVE, buf, -1, -1, err); - clac(); - - return err; -} - /* * Restore xstate from user space xsave area. */ diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 23f1691670b66..c8f5ff58578ed 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -117,23 +117,6 @@ static inline int save_xstate_epilog(void __user *buf, int ia32_frame) return err; } - -static inline int copy_fpregs_to_sigframe(struct xregs_state __user *buf) -{ - int err; - - if (use_xsave()) - err = copy_xregs_to_user(buf); - else if (use_fxsr()) - err = copy_fxregs_to_user((struct fxregs_state __user *) buf); - else - err = copy_fregs_to_user((struct fregs_state __user *) buf); - - if (unlikely(err) && __clear_user(buf, fpu_user_xstate_size)) - err = -EFAULT; - return err; -} - /* * Save the fpu, extended register state to the user signal frame. * @@ -172,27 +155,20 @@ int copy_fpstate_to_sigframe(void __user *buf, void __user *buf_fx, int size) sizeof(struct user_i387_ia32_struct), NULL, (struct _fpstate_32 __user *) buf) ? -1 : 1; - if (fpu->initialized || using_compacted_format()) { - /* Save the live register state to the user directly. */ - if (copy_fpregs_to_sigframe(buf_fx)) - return -1; - /* Update the thread's fxstate to save the fsave header. */ - if (ia32_fxstate) - copy_fxregs_to_kernel(fpu); + /* Update the thread's fxstate to save the fsave header. */ + if (ia32_fxstate) { + copy_fxregs_to_kernel(fpu); } else { - /* - * It is a *bug* if kernel uses compacted-format for xsave - * area and we copy it out directly to a signal frame. It - * should have been handled above by saving the registers - * directly. - */ - if (boot_cpu_has(X86_FEATURE_XSAVES)) { - WARN_ONCE(1, "x86/fpu: saving compacted-format xsave area to a signal frame!\n"); - return -1; - } + copy_fpregs_to_fpstate(fpu); + fpregs_deactivate(fpu); + } + if (using_compacted_format()) { + copy_xstate_to_user(buf_fx, xsave, 0, size); + } else { fpstate_sanitize_xstate(fpu); - if (__copy_to_user(buf_fx, xsave, fpu_user_xstate_size)) + size = fpu_user_xstate_size; + if (__copy_to_user(buf_fx, xsave, size)) return -1; } From patchwork Thu Oct 4 14:05:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626217 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C7C8E15E2 for ; Thu, 4 Oct 2018 14:06:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B8E192916D for ; Thu, 4 Oct 2018 14:06:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AC842291BC; Thu, 4 Oct 2018 14:06:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5455D2916D for ; Thu, 4 Oct 2018 14:06:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727836AbeJDU7i (ORCPT ); Thu, 4 Oct 2018 16:59:38 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36323 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727790AbeJDU7h (ORCPT ); Thu, 4 Oct 2018 16:59:37 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GU-0002ic-ED; Thu, 04 Oct 2018 16:06:06 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 09/11] x86/entry: add TIF_LOAD_FPU Date: Thu, 4 Oct 2018 16:05:45 +0200 Message-Id: <20181004140547.13014-10-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add TIF_LOAD_FPU. This is reserved for loading the FPU registers before returning to userpace. This flag must not be set for systems without a FPU. It is introduced now, so we can add code handling it now before adding the main feature. Signed-off-by: Sebastian Andrzej Siewior --- arch/x86/include/asm/thread_info.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thread_info.h index cd6920674b905..cd910fd8c8c4c 100644 --- a/arch/x86/include/asm/thread_info.h +++ b/arch/x86/include/asm/thread_info.h @@ -83,6 +83,7 @@ struct thread_info { #define TIF_SYSCALL_EMU 6 /* syscall emulation active */ #define TIF_SYSCALL_AUDIT 7 /* syscall auditing active */ #define TIF_SECCOMP 8 /* secure computing */ +#define TIF_LOAD_FPU 10 /* load FPU on return to userspace */ #define TIF_USER_RETURN_NOTIFY 11 /* notify kernel of userspace return */ #define TIF_UPROBE 12 /* breakpointed or singlestepping */ #define TIF_PATCH_PENDING 13 /* pending live patching update */ @@ -110,6 +111,7 @@ struct thread_info { #define _TIF_SYSCALL_EMU (1 << TIF_SYSCALL_EMU) #define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT) #define _TIF_SECCOMP (1 << TIF_SECCOMP) +#define _TIF_LOAD_FPU (1 << TIF_LOAD_FPU) #define _TIF_USER_RETURN_NOTIFY (1 << TIF_USER_RETURN_NOTIFY) #define _TIF_UPROBE (1 << TIF_UPROBE) #define _TIF_PATCH_PENDING (1 << TIF_PATCH_PENDING) From patchwork Thu Oct 4 14:05:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626215 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3024A15E2 for ; Thu, 4 Oct 2018 14:06:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 20C7D2916D for ; Thu, 4 Oct 2018 14:06:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1426C291BC; Thu, 4 Oct 2018 14:06:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AA2C92916D for ; Thu, 4 Oct 2018 14:06:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727950AbeJDU7k (ORCPT ); Thu, 4 Oct 2018 16:59:40 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36331 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727356AbeJDU7j (ORCPT ); Thu, 4 Oct 2018 16:59:39 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GV-0002ic-8f; Thu, 04 Oct 2018 16:06:07 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 10/11] x86/fpu: prepare copy_fpstate_to_sigframe for TIF_LOAD_FPU Date: Thu, 4 Oct 2018 16:05:46 +0200 Message-Id: <20181004140547.13014-11-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Rik van Riel If TIF_LOAD_FPU is set, then the registers are saved (not loaded). In that case we skip the saving part. Signed-off-by: Rik van Riel Signed-off-by: Sebastian Andrzej Siewior --- arch/x86/kernel/fpu/signal.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index c8f5ff58578ed..979dcd1ed82e0 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -155,13 +155,17 @@ int copy_fpstate_to_sigframe(void __user *buf, void __user *buf_fx, int size) sizeof(struct user_i387_ia32_struct), NULL, (struct _fpstate_32 __user *) buf) ? -1 : 1; - /* Update the thread's fxstate to save the fsave header. */ - if (ia32_fxstate) { - copy_fxregs_to_kernel(fpu); - } else { - copy_fpregs_to_fpstate(fpu); - fpregs_deactivate(fpu); + __fpregs_changes_begin(); + if (!test_thread_flag(TIF_LOAD_FPU)) { + /* Update the thread's fxstate to save the fsave header. */ + if (ia32_fxstate) { + copy_fxregs_to_kernel(fpu); + } else { + copy_fpregs_to_fpstate(fpu); + fpregs_deactivate(fpu); + } } + __fpregs_changes_end(); if (using_compacted_format()) { copy_xstate_to_user(buf_fx, xsave, 0, size); From patchwork Thu Oct 4 14:05:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 10626213 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3F4181515 for ; Thu, 4 Oct 2018 14:06:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2E33D2916D for ; Thu, 4 Oct 2018 14:06:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 21851291BC; Thu, 4 Oct 2018 14:06:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC2012916D for ; Thu, 4 Oct 2018 14:06:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727974AbeJDU7l (ORCPT ); Thu, 4 Oct 2018 16:59:41 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:36339 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727380AbeJDU7k (ORCPT ); Thu, 4 Oct 2018 16:59:40 -0400 Received: from localhost ([127.0.0.1] helo=bazinga.breakpoint.cc) by Galois.linutronix.de with esmtp (Exim 4.80) (envelope-from ) id 1g84GW-0002ic-3i; Thu, 04 Oct 2018 16:06:08 +0200 From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: x86@kernel.org, Andy Lutomirski , Paolo Bonzini , =?utf-8?b?UmFkaW0gS3LEjW3DocWZ?= , kvm@vger.kernel.org, "Jason A. Donenfeld" , Rik van Riel , Dave Hansen , Sebastian Andrzej Siewior Subject: [PATCH 11/11] x86/fpu: defer FPU state load until return to userspace Date: Thu, 4 Oct 2018 16:05:47 +0200 Message-Id: <20181004140547.13014-12-bigeasy@linutronix.de> X-Mailer: git-send-email 2.19.0 In-Reply-To: <20181004140547.13014-1-bigeasy@linutronix.de> References: <20181004140547.13014-1-bigeasy@linutronix.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Rik van Riel Defer loading of FPU state until return to userspace. This gives the kernel the potential to skip loading FPU state for tasks that stay in kernel mode, or for tasks that end up with repeated invocations of kernel_fpu_begin. It also increases the chances that a task's FPU state will remain valid in the FPU registers until it is scheduled back in, allowing us to skip restoring that task's FPU state altogether. The __fpregs_changes_{begin|end}() section ensures that the register remain unchanged. Otherwise a context switch or a BH could save the registers to its FPU context and processor's FPU register would remain random. fpu__restore() has one user so I pulled that preempt_disable() part into fpu__restore(). While the function did *load* the registers, it now just makes sure that they are loaded on return to userland. KVM swaps the host/guest register on enry/exit path. I kept the flow as is. First it ensures that the registers are loaded and then saves the current (host) state before it loads the guest's register. Before entring the guest, it ensures that the register are still loaded. Signed-off-by: Rik van Riel Signed-off-by: Sebastian Andrzej Siewior --- arch/x86/entry/common.c | 9 +++ arch/x86/include/asm/fpu/api.h | 11 +++ arch/x86/include/asm/fpu/internal.h | 25 ++++--- arch/x86/include/asm/trace/fpu.h | 5 +- arch/x86/kernel/fpu/core.c | 108 ++++++++++++++++++++-------- arch/x86/kernel/fpu/signal.c | 3 - arch/x86/kernel/process.c | 2 +- arch/x86/kernel/process_32.c | 7 +- arch/x86/kernel/process_64.c | 7 +- arch/x86/kvm/x86.c | 18 +++-- 10 files changed, 143 insertions(+), 52 deletions(-) diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c index 3b2490b819181..3dad5c3b335eb 100644 --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -31,6 +31,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -196,6 +197,14 @@ __visible inline void prepare_exit_to_usermode(struct pt_regs *regs) if (unlikely(cached_flags & EXIT_TO_USERMODE_LOOP_FLAGS)) exit_to_usermode_loop(regs, cached_flags); + /* Reload ti->flags; we may have rescheduled above. */ + cached_flags = READ_ONCE(ti->flags); + + if (unlikely(cached_flags & _TIF_LOAD_FPU)) + switch_fpu_return(); + else + fpregs_is_state_consistent(); + #ifdef CONFIG_COMPAT /* * Compat syscalls set TS_COMPAT. Make sure we clear it before diff --git a/arch/x86/include/asm/fpu/api.h b/arch/x86/include/asm/fpu/api.h index a9caac9d4a729..e3077860f7333 100644 --- a/arch/x86/include/asm/fpu/api.h +++ b/arch/x86/include/asm/fpu/api.h @@ -27,6 +27,17 @@ extern void kernel_fpu_begin(void); extern void kernel_fpu_end(void); extern bool irq_fpu_usable(void); +#ifdef CONFIG_X86_DEBUG_FPU +extern void fpregs_is_state_consistent(void); +#else +static inline void fpregs_is_state_consistent(void) { } +#endif + +/* + * Load the task FPU state before returning to userspace. + */ +extern void switch_fpu_return(void); + /* * Query the presence of one or more xfeatures. Works on any legacy CPU as well. * diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h index df8816be3efdd..346f8057ecd7b 100644 --- a/arch/x86/include/asm/fpu/internal.h +++ b/arch/x86/include/asm/fpu/internal.h @@ -32,7 +32,7 @@ extern void fpu__save(struct fpu *fpu); extern void fpu__restore(struct fpu *fpu); extern int fpu__restore_sig(void __user *buf, int ia32_frame); extern void fpu__drop(struct fpu *fpu); -extern int fpu__copy(struct fpu *dst_fpu, struct fpu *src_fpu); +extern int fpu__copy(struct task_struct *dst, struct task_struct *src); extern void fpu__clear(struct fpu *fpu); extern int fpu__exception_code(struct fpu *fpu, int trap_nr); extern int dump_fpu(struct pt_regs *ptregs, struct user_i387_struct *fpstate); @@ -473,21 +473,30 @@ static inline void fpregs_activate(struct fpu *fpu) /* * Load the FPU state for the current task. Call with preemption disabled. */ -static inline void __fpregs_load_activate(struct fpu *fpu, int cpu) +static inline void __fpregs_load_activate(void) { + struct fpu *fpu = ¤t->thread.fpu; + int cpu = smp_processor_id(); + if (!fpregs_state_valid(fpu, cpu)) copy_kernel_to_fpregs(&fpu->state); fpregs_activate(fpu); + fpu->last_cpu = cpu; + clear_thread_flag(TIF_LOAD_FPU); } +void fpregs_load_activate(void); + static inline void __fpregs_changes_begin(void) { preempt_disable(); + local_bh_disable(); } static inline void __fpregs_changes_end(void) { preempt_enable(); + local_bh_enable(); } /* @@ -498,8 +507,8 @@ static inline void __fpregs_changes_end(void) * - switch_fpu_prepare() saves the old state. * This is done within the context of the old process. * - * - switch_fpu_finish() restores the new state as - * necessary. + * - switch_fpu_finish() sets TIF_LOAD_FPU; the floating point state + * will get loaded on return to userspace, or when the kernel needs it. */ static inline void switch_fpu_prepare(struct fpu *old_fpu, int cpu) @@ -521,10 +530,10 @@ switch_fpu_prepare(struct fpu *old_fpu, int cpu) */ /* - * Set up the userspace FPU context for the new task, if the task - * has used the FPU. + * Load PKRU from the FPU context if available. Delay loading the loading of the + * complete FPU state until the return to userland. */ -static inline void switch_fpu_finish(struct fpu *new_fpu, int cpu) +static inline void switch_fpu_finish(struct fpu *new_fpu) { bool load_fpu; @@ -545,7 +554,7 @@ static inline void switch_fpu_finish(struct fpu *new_fpu, int cpu) #endif if (!load_fpu) return; - __fpregs_load_activate(new_fpu, cpu); + set_thread_flag(TIF_LOAD_FPU); } /* diff --git a/arch/x86/include/asm/trace/fpu.h b/arch/x86/include/asm/trace/fpu.h index 069c04be15076..ec3be1c9da7ec 100644 --- a/arch/x86/include/asm/trace/fpu.h +++ b/arch/x86/include/asm/trace/fpu.h @@ -14,6 +14,7 @@ DECLARE_EVENT_CLASS(x86_fpu, TP_STRUCT__entry( __field(struct fpu *, fpu) __field(bool, initialized) + __field(bool, load_fpu) __field(u64, xfeatures) __field(u64, xcomp_bv) ), @@ -21,14 +22,16 @@ DECLARE_EVENT_CLASS(x86_fpu, TP_fast_assign( __entry->fpu = fpu; __entry->initialized = fpu->initialized; + __entry->load_fpu = test_thread_flag(TIF_LOAD_FPU); if (boot_cpu_has(X86_FEATURE_OSXSAVE)) { __entry->xfeatures = fpu->state.xsave.header.xfeatures; __entry->xcomp_bv = fpu->state.xsave.header.xcomp_bv; } ), - TP_printk("x86/fpu: %p initialized: %d xfeatures: %llx xcomp_bv: %llx", + TP_printk("x86/fpu: %p initialized: %d load: %d xfeatures: %llx xcomp_bv: %llx", __entry->fpu, __entry->initialized, + __entry->load_fpu, __entry->xfeatures, __entry->xcomp_bv ) diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 72cd2e2a07194..c757fd1a8440d 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -101,14 +101,15 @@ void __kernel_fpu_begin(void) kernel_fpu_disable(); - if (fpu->initialized) { + __cpu_invalidate_fpregs_state(); + + if (!test_thread_flag(TIF_LOAD_FPU)) { + set_thread_flag(TIF_LOAD_FPU); /* * Ignore return value -- we don't care if reg state * is clobbered. */ copy_fpregs_to_fpstate(fpu); - } else { - __cpu_invalidate_fpregs_state(); } } EXPORT_SYMBOL(__kernel_fpu_begin); @@ -117,8 +118,7 @@ void __kernel_fpu_end(void) { struct fpu *fpu = ¤t->thread.fpu; - if (fpu->initialized) - copy_kernel_to_fpregs(&fpu->state); + switch_fpu_finish(fpu); kernel_fpu_enable(); } @@ -147,15 +147,15 @@ void fpu__save(struct fpu *fpu) { WARN_ON_FPU(fpu != ¤t->thread.fpu); - preempt_disable(); + __fpregs_changes_begin(); trace_x86_fpu_before_save(fpu); - if (fpu->initialized) { + if (fpu->initialized && !test_thread_flag(TIF_LOAD_FPU)) { if (!copy_fpregs_to_fpstate(fpu)) { copy_kernel_to_fpregs(&fpu->state); } } trace_x86_fpu_after_save(fpu); - preempt_enable(); + __fpregs_changes_end(); } EXPORT_SYMBOL_GPL(fpu__save); @@ -188,8 +188,11 @@ void fpstate_init(union fpregs_state *state) } EXPORT_SYMBOL_GPL(fpstate_init); -int fpu__copy(struct fpu *dst_fpu, struct fpu *src_fpu) +int fpu__copy(struct task_struct *dst, struct task_struct *src) { + struct fpu *dst_fpu = &dst->thread.fpu; + struct fpu *src_fpu = &src->thread.fpu; + dst_fpu->last_cpu = -1; if (!src_fpu->initialized || !static_cpu_has(X86_FEATURE_FPU)) @@ -204,16 +207,23 @@ int fpu__copy(struct fpu *dst_fpu, struct fpu *src_fpu) memset(&dst_fpu->state.xsave, 0, fpu_kernel_xstate_size); /* - * Save current FPU registers directly into the child + * If the FPU registers are not loaded just memcpy() the state. + * Otherwise save current FPU registers directly into the child * FPU context, without any memory-to-memory copying. * * ( The function 'fails' in the FNSAVE case, which destroys - * register contents so we have to copy them back. ) + * register contents so we have to load them back. ) */ - if (!copy_fpregs_to_fpstate(dst_fpu)) { - memcpy(&src_fpu->state, &dst_fpu->state, fpu_kernel_xstate_size); - copy_kernel_to_fpregs(&src_fpu->state); - } + __fpregs_changes_begin(); + if (test_thread_flag(TIF_LOAD_FPU)) + memcpy(&dst_fpu->state, &src_fpu->state, fpu_kernel_xstate_size); + + else if (!copy_fpregs_to_fpstate(dst_fpu)) + copy_kernel_to_fpregs(&dst_fpu->state); + + __fpregs_changes_end(); + + set_tsk_thread_flag(dst, TIF_LOAD_FPU); trace_x86_fpu_copy_src(src_fpu); trace_x86_fpu_copy_dst(dst_fpu); @@ -236,6 +246,7 @@ void fpu__initialize(struct fpu *fpu) trace_x86_fpu_activate_state(fpu); /* Safe to do for the current task: */ fpu->initialized = 1; + set_thread_flag(TIF_LOAD_FPU); } } EXPORT_SYMBOL_GPL(fpu__initialize); @@ -306,26 +317,18 @@ void fpu__prepare_write(struct fpu *fpu) } /* - * 'fpu__restore()' is called to copy FPU registers from - * the FPU fpstate to the live hw registers and to activate - * access to the hardware registers, so that FPU instructions - * can be used afterwards. - * - * Must be called with kernel preemption disabled (for example - * with local interrupts disabled, as it is in the case of - * do_device_not_available()). + * 'fpu__restore()' is called to ensure that the FPU registers in fpstate + * are loaded on return to userspace. */ void fpu__restore(struct fpu *fpu) { - fpu__initialize(fpu); + WARN_ON_FPU(fpu != ¤t->thread.fpu); - /* Avoid __kernel_fpu_begin() right after fpregs_activate() */ - kernel_fpu_disable(); trace_x86_fpu_before_restore(fpu); - fpregs_activate(fpu); - copy_kernel_to_fpregs(&fpu->state); + __fpu_invalidate_fpregs_state(fpu); + fpu->initialized = 1; + set_thread_flag(TIF_LOAD_FPU); trace_x86_fpu_after_restore(fpu); - kernel_fpu_enable(); } EXPORT_SYMBOL_GPL(fpu__restore); @@ -400,6 +403,53 @@ void fpu__clear(struct fpu *fpu) } } +/* + * Load FPU context before returning to userspace. + */ +void switch_fpu_return(void) +{ + if (!static_cpu_has(X86_FEATURE_FPU)) + return; + + /* + * We should never return to user space without the task's + * own FPU contents loaded into the registers. That makes it + * a bug to not have the task's FPU state set up. + */ + WARN_ON_FPU(!current->thread.fpu.initialized); + + __fpregs_load_activate(); +} +EXPORT_SYMBOL_GPL(switch_fpu_return); + +#ifdef CONFIG_X86_DEBUG_FPU +/* + * If current FPU state according to its tracking (loaded FPU ctx on this CPU) + * is not valid then we must have TIF_LOAD_FPU set so the context is loaded on + * return to userland. + */ +void fpregs_is_state_consistent(void) +{ + struct fpu *fpu = ¤t->thread.fpu; + + if (!fpu->initialized) + return; + if (test_thread_flag(TIF_LOAD_FPU)) + return; + WARN_ON_FPU(!fpregs_state_valid(fpu, smp_processor_id())); +} +EXPORT_SYMBOL_GPL(fpregs_is_state_consistent); +#endif + +void fpregs_load_activate(void) +{ + if (test_thread_flag(TIF_LOAD_FPU)) + __fpregs_load_activate(); + else + fpregs_is_state_consistent(); +} +EXPORT_SYMBOL_GPL(fpregs_load_activate); + /* * x87 math exception handling: */ diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c index 979dcd1ed82e0..45d2f165b47ac 100644 --- a/arch/x86/kernel/fpu/signal.c +++ b/arch/x86/kernel/fpu/signal.c @@ -325,10 +325,7 @@ static int __fpu__restore_sig(void __user *buf, void __user *buf_fx, int size) sanitize_restored_xstate(tsk, &env, xfeatures, fx_only); } - fpu->initialized = 1; - preempt_disable(); fpu__restore(fpu); - preempt_enable(); return err; } else { diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index c93fcfdf16734..cd7105fb92bfc 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -96,7 +96,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) dst->thread.vm86 = NULL; #endif - return fpu__copy(&dst->thread.fpu, &src->thread.fpu); + return fpu__copy(dst, src); } /* diff --git a/arch/x86/kernel/process_32.c b/arch/x86/kernel/process_32.c index 5046a3c9dec2f..a65f8ce36379b 100644 --- a/arch/x86/kernel/process_32.c +++ b/arch/x86/kernel/process_32.c @@ -236,7 +236,8 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) /* never put a printk in __switch_to... printk() calls wake_up*() indirectly */ - switch_fpu_prepare(prev_fpu, cpu); + if (prev_fpu->initialized && !test_thread_flag(TIF_LOAD_FPU)) + switch_fpu_prepare(prev_fpu, cpu); /* * Save away %gs. No need to save %fs, as it was saved on the @@ -297,10 +298,10 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) if (prev->gs | next->gs) lazy_load_gs(next->gs); - switch_fpu_finish(next_fpu, cpu); - this_cpu_write(current_task, next_p); + switch_fpu_finish(next_fpu); + /* Load the Intel cache allocation PQR MSR. */ intel_rdt_sched_in(); diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index ea5ea850348da..66b763f3da6a0 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -427,7 +427,8 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) WARN_ON_ONCE(IS_ENABLED(CONFIG_DEBUG_ENTRY) && this_cpu_read(irq_count) != -1); - switch_fpu_prepare(prev_fpu, cpu); + if (prev_fpu->initialized && !test_thread_flag(TIF_LOAD_FPU)) + switch_fpu_prepare(prev_fpu, cpu); /* We must save %fs and %gs before load_TLS() because * %fs and %gs may be cleared by load_TLS(). @@ -478,8 +479,6 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) load_seg_legacy(prev->gsindex, prev->gsbase, next->gsindex, next->gsbase, GS); - switch_fpu_finish(next_fpu, cpu); - /* * Switch the PDA and FPU contexts. */ @@ -489,6 +488,8 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) /* Reload sp0. */ update_task_stack(next_p); + switch_fpu_finish(next_fpu); + /* * Now maybe reload the debug registers and handle I/O bitmaps */ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index edbf00ec56b34..2a9f35e8bb81e 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -7592,6 +7592,11 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) wait_lapic_expire(vcpu); guest_enter_irqoff(); + if (test_thread_flag(TIF_LOAD_FPU)) + switch_fpu_return(); + else + fpregs_is_state_consistent(); + if (unlikely(vcpu->arch.switch_db_regs)) { set_debugreg(0, 7); set_debugreg(vcpu->arch.eff_db[0], 0); @@ -7851,22 +7856,27 @@ static int complete_emulated_mmio(struct kvm_vcpu *vcpu) /* Swap (qemu) user FPU context for the guest FPU context. */ static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu) { - preempt_disable(); + __fpregs_changes_begin(); + fpregs_load_activate(); + copy_fpregs_to_fpstate(&vcpu->arch.user_fpu); + /* PKRU is separately restored in kvm_x86_ops->run. */ __copy_kernel_to_fpregs(&vcpu->arch.guest_fpu.state, ~XFEATURE_MASK_PKRU); - preempt_enable(); + __fpregs_changes_end(); trace_kvm_fpu(1); } /* When vcpu_run ends, restore user space FPU context. */ static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu) { - preempt_disable(); + __fpregs_changes_begin(); + fpregs_load_activate(); + copy_fpregs_to_fpstate(&vcpu->arch.guest_fpu); copy_kernel_to_fpregs(&vcpu->arch.user_fpu.state); - preempt_enable(); + __fpregs_changes_end(); ++vcpu->stat.fpu_reload; trace_kvm_fpu(0); }