From patchwork Fri Oct 20 20:44:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 13431105 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACB72C0032E for ; Fri, 20 Oct 2023 20:45:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230403AbjJTUpB (ORCPT ); Fri, 20 Oct 2023 16:45:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229983AbjJTUpA (ORCPT ); Fri, 20 Oct 2023 16:45:00 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95A26E8; Fri, 20 Oct 2023 13:44:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834698; x=1729370698; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=xzkWuakHUz1psAcA/doANbYYppEAuf8OEljg8J5mWNg=; b=FoJYlN+MSoMMbtFKT8O1yBnhE4HHU4tl+Jz+nluseou204nITad+MgMF qLSwEhFvebYakO9/PNM/k6Q8wl9pG/UNOb7pXBmRPqd/g+U9wZJ4MxD93 pkmByX0qXXZluiZl5R3ZdwgbdtO7hEdl02btHiFYyodLRzcO9RNs1IkZz syPsl81eZNeps8IkgLxf9Ht8e7oZ3k4/9OiB1Y6R/DMOnRphxgZMZY/MA 1YahkF/PiOEYNBuCKfd91i/lqClESprZbmpzkuEUZnTrS1d71BDNfMNTt qkpkwiXY4Q7ib8Fs2W32CCQ0xe/kekgYnYPirQPK1LjctRmmXjxiqT/OS Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="371640195" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="371640195" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:44:58 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="931117471" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="931117471" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:44:57 -0700 Date: Fri, 20 Oct 2023 13:44:57 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta , Alyssa Milburn Subject: [PATCH 1/6] x86/bugs: Add asm helpers for executing VERW Message-ID: <20231020-delay-verw-v1-1-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org MDS mitigation requires clearing the CPU buffers before returning to user. This needs to be done late in the exit-to-user path. Current location of VERW leaves a possibility of kernel data ending up in CPU buffers for memory accesses done after VERW such as: 1. Kernel data accessed by an NMI between VERW and return-to-user can remain in CPU buffers ( since NMI returning to kernel does not execute VERW to clear CPU buffers. 2. Alyssa reported that after VERW is executed, CONFIG_GCC_PLUGIN_STACKLEAK=y scrubs the stack used by a system call. Memory accesses during stack scrubbing can move kernel stack contents into CPU buffers. 3. When caller saved registers are restored after a return from function executing VERW, the kernel stack accesses can remain in CPU buffers(since they occur after VERW). To fix this VERW needs to be moved very late in exit-to-user path. In preparation for moving VERW to entry/exit asm code, create macros that can be used in asm. Also make them depend on a new feature flag X86_FEATURE_USER_CLEAR_CPU_BUF. Reported-by: Alyssa Milburn Signed-off-by: Pawan Gupta --- arch/x86/include/asm/cpufeatures.h | 2 +- arch/x86/include/asm/nospec-branch.h | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 58cb9495e40f..3f018dfedd5f 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -308,10 +308,10 @@ #define X86_FEATURE_SMBA (11*32+21) /* "" Slow Memory Bandwidth Allocation */ #define X86_FEATURE_BMEC (11*32+22) /* "" Bandwidth Monitoring Event Configuration */ #define X86_FEATURE_USER_SHSTK (11*32+23) /* Shadow stack support for user mode applications */ - #define X86_FEATURE_SRSO (11*32+24) /* "" AMD BTB untrain RETs */ #define X86_FEATURE_SRSO_ALIAS (11*32+25) /* "" AMD BTB untrain RETs through aliasing */ #define X86_FEATURE_IBPB_ON_VMEXIT (11*32+26) /* "" Issue an IBPB only on VMEXIT */ +#define X86_FEATURE_USER_CLEAR_CPU_BUF (11*32+27) /* "" Clear CPU buffers before returning to user */ /* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */ #define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */ diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index c55cc243592e..e1b623a27e1b 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -111,6 +111,24 @@ #define RESET_CALL_DEPTH_FROM_CALL #endif +/* + * Macro to execute VERW instruction to mitigate transient data sampling + * attacks such as MDS. On affected systems a microcode update overloaded VERW + * instruction to also clear the CPU buffers. + * + * Note: Only the memory operand variant of VERW clears the CPU buffers. To + * handle the case when VERW is executed after user registers are restored, use + * RIP to point the memory operand to a part NOPL instruction that contains + * __KERNEL_DS. + */ +#define __EXEC_VERW(m) verw _ASM_RIP(m) + +#define EXEC_VERW \ + __EXEC_VERW(551f); \ + /* nopl __KERNEL_DS(%rax) */ \ + .byte 0x0f, 0x1f, 0x80, 0x00, 0x00; \ +551: .word __KERNEL_DS; \ + /* * Fill the CPU return stack buffer. * @@ -329,6 +347,13 @@ #endif .endm +/* Clear CPU buffers before returning to user */ +.macro USER_CLEAR_CPU_BUFFERS + ALTERNATIVE "jmp .Lskip_verw_\@;", "", X86_FEATURE_USER_CLEAR_CPU_BUF + EXEC_VERW +.Lskip_verw_\@: +.endm + #else /* __ASSEMBLY__ */ #define ANNOTATE_RETPOLINE_SAFE \ From patchwork Fri Oct 20 20:45:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 13431106 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 113C3C0032E for ; Fri, 20 Oct 2023 20:45:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230405AbjJTUpK (ORCPT ); Fri, 20 Oct 2023 16:45:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231160AbjJTUpI (ORCPT ); Fri, 20 Oct 2023 16:45:08 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA248D72; Fri, 20 Oct 2023 13:45:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834704; x=1729370704; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=uGr8ClT2h4Zv+dGb/ooumBjLtEvJ/qtZo0hY1GnJHiw=; b=JtQId4Tq+iQT8NvWsje/7gQ+4wjmiFSQjQoSYO3NTHzd+QxGHX2E4WYI femvCycQI5en2W0y0weDxNYO9IStc15L2wbz5a8mtpUja2R15r/IOGqJM FWgiWue0E/yxhsNcDLd1YC4lgNAJ3tCO8eekbtpNfReSDNKFL6gYSn7O+ 8jj4E3GQQRsdsqHasxx8I/cwY6nV5khn990lJ3qTVpPvb2QNF+gNTh+RJ 7z3LMW5LTxWCI2uqal+hxPNCQlW1RWiDcpumRNzdwsAv1FFZBOa+EYQKu AjhJ5cDllDFVcH0WdcebcdhLvf+6eEeSQ/ul4MWOguoCiM6JddsF5JQbU Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="453048206" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="453048206" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="5217189" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by smtpauth.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:43:52 -0700 Date: Fri, 20 Oct 2023 13:45:03 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta , Dave Hansen Subject: [PATCH 2/6] x86/entry_64: Add VERW just before userspace transition Message-ID: <20231020-delay-verw-v1-2-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Mitigation for MDS is to use VERW instruction to clear any secrets in CPU Buffers. Any memory accesses after VERW execution can still remain in CPU buffers. It is safer to execute VERW late in return to user path to minimize the window in which kernel data can end up in CPU buffers. There are not many kernel secrets to be had after SWITCH_TO_USER_CR3. Add support for deploying VERW mitigation after user register state is restored. This helps minimize the chances of kernel data ending up into CPU buffers after executing VERW. Note that the mitigation at the new location is not yet enabled. Suggested-by: Dave Hansen Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_64.S | 14 ++++++++++++++ arch/x86/entry/entry_64_compat.S | 2 ++ 2 files changed, 16 insertions(+) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 43606de22511..e72ac30f0714 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -223,6 +223,8 @@ syscall_return_via_sysret: SYM_INNER_LABEL(entry_SYSRETQ_unsafe_stack, SYM_L_GLOBAL) ANNOTATE_NOENDBR swapgs + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS sysretq SYM_INNER_LABEL(entry_SYSRETQ_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR @@ -663,6 +665,10 @@ SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL) /* Restore RDI. */ popq %rdi swapgs + + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS + jmp .Lnative_iret @@ -774,6 +780,9 @@ native_irq_return_ldt: */ popq %rax /* Restore user RAX */ + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS + /* * RSP now points to an ordinary IRET frame, except that the page * is read-only and RSP[31:16] are preloaded with the userspace @@ -1502,6 +1511,9 @@ nmi_restore: std movq $0, 5*8(%rsp) /* clear "NMI executing" */ + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS + /* * iretq reads the "iret" frame and exits the NMI stack in a * single instruction. We are returning to kernel mode, so this @@ -1520,6 +1532,8 @@ SYM_CODE_START(ignore_sysret) UNWIND_HINT_END_OF_STACK ENDBR mov $-ENOSYS, %eax + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS sysretl SYM_CODE_END(ignore_sysret) #endif diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_compat.S index 70150298f8bd..d2ccd9148239 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -271,6 +271,8 @@ SYM_INNER_LABEL(entry_SYSRETL_compat_unsafe_stack, SYM_L_GLOBAL) xorl %r9d, %r9d xorl %r10d, %r10d swapgs + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS sysretl SYM_INNER_LABEL(entry_SYSRETL_compat_end, SYM_L_GLOBAL) ANNOTATE_NOENDBR From patchwork Fri Oct 20 20:45:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 13431107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9F122C001DF for ; Fri, 20 Oct 2023 20:45:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232318AbjJTUpU (ORCPT ); Fri, 20 Oct 2023 16:45:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230408AbjJTUpP (ORCPT ); Fri, 20 Oct 2023 16:45:15 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D711D73; Fri, 20 Oct 2023 13:45:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834711; x=1729370711; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=0qW8gkQ9cQn9nJIAFp1D4GChim4j3ri4y26baGhJi2U=; b=TgfSmSbN2UWoYOR68QBC3ieQZ05CW1/XqLHaJxayGBCoJn/5/ybDPij/ EYG4rXAafabNr1tzgTAlVhftdr7xGEwHdld8P7HZBI9rg0vdFPTs+XkP+ 4dYd+5V1U0XAO0ZsU0LzeGILbka7tIEj17kwG0izuM0XibxtlSQlVlMnP szjlDwHWh9XBMCE6VdHcEZUKJ17aRpCfHp8GdfuKuH23km6dyy6yXqVVP un76a1JOnAPFcfe+GXBCDZkEiLesavYqajCpNi1MwUE102s5vROJ7S3mx gzaYTnX1dY0T3aoJ0x0JAFSOInqsiPsj5FZG+RouyItpLFdxnucZgAzpz w==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="371640238" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="371640238" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="931117494" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="931117494" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:09 -0700 Date: Fri, 20 Oct 2023 13:45:09 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH 3/6] x86/entry_32: Add VERW just before userspace transition Message-ID: <20231020-delay-verw-v1-3-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As done for entry_64, add support for executing VERW late in exit to user path for 32-bit mode. Signed-off-by: Pawan Gupta --- arch/x86/entry/entry_32.S | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 6e6af42e044a..bbf77d2aab2e 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -886,6 +886,9 @@ SYM_FUNC_START(entry_SYSENTER_32) popfl popl %eax + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS + /* * Return back to the vDSO, which will pop ecx and edx. * Don't bother with DS and ES (they already contain __USER_DS). @@ -954,6 +957,9 @@ restore_all_switch_stack: /* Restore user state */ RESTORE_REGS pop=4 # skip orig_eax/error_code + + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS .Lirq_return: /* * ARCH_HAS_MEMBARRIER_SYNC_CORE rely on IRET core serialization @@ -1146,6 +1152,8 @@ SYM_CODE_START(asm_exc_nmi) /* Not on SYSENTER stack. */ call exc_nmi + /* Mitigate CPU data sampling attacks .e.g. MDS */ + USER_CLEAR_CPU_BUFFERS jmp .Lnmi_return .Lnmi_from_sysenter_stack: From patchwork Fri Oct 20 20:45:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 13431108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94A7DC0032E for ; Fri, 20 Oct 2023 20:45:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232859AbjJTUpf (ORCPT ); Fri, 20 Oct 2023 16:45:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231354AbjJTUp1 (ORCPT ); Fri, 20 Oct 2023 16:45:27 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D92610F0; Fri, 20 Oct 2023 13:45:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834718; x=1729370718; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=mbuqw2cUwhH3Vh8gF1I6goIWRvTTaGe6ZC14bjh7nw8=; b=DhHJxzTAGENVxsznas6zwauTC7DN77uoWtPYE5NUabvSohsCIctjepGm H7Nws6PDyrKxdhlxgy4Ym6ectHNtR+EFwAyaP9FC7ewp5Wxak/yCFSJET e3ZHJ2/qYC03Y3m73LvWNyWnMz/LZ4EZoO2mADKNzKsOfGxjbCsT+go02 C5EPek/QUdo5zOYc3tV2JPuWathnX6D+UthtHwq5brVfq68xYVOMTqBAz 5GiNzkfkQpuTs4Uw/lUmA57tKOm2agvw0fFQDhdUI2VqdLvKnbhj0GDZW m7FZ5Qzoutnbg8lYe2q3ZXcONx4C/2v1BMPjjUP0G0ugJq464ppDCsXSq g==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="365909393" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="365909393" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="848201701" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="848201701" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:16 -0700 Date: Fri, 20 Oct 2023 13:45:15 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH 4/6] x86/bugs: Use ALTERNATIVE() instead of mds_user_clear static key Message-ID: <20231020-delay-verw-v1-4-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The VERW mitigation at exit-to-user is enabled via a static branch mds_user_clear. This static branch is never toggled after boot, and can be safely replaced with an ALTERNATIVE() which is convenient to use in asm. Switch to ALTERNATIVE() to use the VERW mitigation late in exit-to-user path. Also remove the now redundant VERW in exc_nmi() and arch_exit_to_user_mode(). Signed-off-by: Pawan Gupta --- Documentation/arch/x86/mds.rst | 20 +++++++++----------- arch/x86/include/asm/entry-common.h | 1 - arch/x86/include/asm/nospec-branch.h | 11 ----------- arch/x86/kernel/cpu/bugs.c | 10 +++++----- arch/x86/kernel/nmi.c | 2 -- arch/x86/kvm/vmx/vmx.c | 2 +- 6 files changed, 15 insertions(+), 31 deletions(-) diff --git a/Documentation/arch/x86/mds.rst b/Documentation/arch/x86/mds.rst index e73fdff62c0a..05090f33f65e 100644 --- a/Documentation/arch/x86/mds.rst +++ b/Documentation/arch/x86/mds.rst @@ -95,6 +95,9 @@ The kernel provides a function to invoke the buffer clearing: mds_clear_cpu_buffers() +Also macro USER_CLEAR_CPU_BUFFERS is meant to be used in ASM late in +exit-to-user path. This macro works for cases where GPRs can't be clobbered. + The mitigation is invoked on kernel/userspace, hypervisor/guest and C-state (idle) transitions. @@ -138,18 +141,13 @@ Mitigation points When transitioning from kernel to user space the CPU buffers are flushed on affected CPUs when the mitigation is not disabled on the kernel - command line. The migitation is enabled through the static key - mds_user_clear. - - The mitigation is invoked in prepare_exit_to_usermode() which covers - all but one of the kernel to user space transitions. The exception - is when we return from a Non Maskable Interrupt (NMI), which is - handled directly in do_nmi(). - - (The reason that NMI is special is that prepare_exit_to_usermode() can - enable IRQs. In NMI context, NMIs are blocked, and we don't want to - enable IRQs with NMIs blocked.) + command line. The migitation is enabled through the feature flag + X86_FEATURE_USER_CLEAR_CPU_BUF. + The mitigation is invoked just before transitioning to userspace after + user registers are restored. This is done to minimize the window in + which kernel data could be accessed after VERW e.g. via an NMI after + VERW. 2. C-State transition ^^^^^^^^^^^^^^^^^^^^^ diff --git a/arch/x86/include/asm/entry-common.h b/arch/x86/include/asm/entry-common.h index ce8f50192ae3..7e523bb3d2d3 100644 --- a/arch/x86/include/asm/entry-common.h +++ b/arch/x86/include/asm/entry-common.h @@ -91,7 +91,6 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, static __always_inline void arch_exit_to_user_mode(void) { - mds_user_clear_cpu_buffers(); amd_clear_divider(); } #define arch_exit_to_user_mode arch_exit_to_user_mode diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index e1b623a27e1b..501c26ecd722 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -595,17 +595,6 @@ static __always_inline void mds_clear_cpu_buffers(void) asm volatile("verw %[ds]" : : [ds] "m" (ds) : "cc"); } -/** - * mds_user_clear_cpu_buffers - Mitigation for MDS and TAA vulnerability - * - * Clear CPU buffers if the corresponding static key is enabled - */ -static __always_inline void mds_user_clear_cpu_buffers(void) -{ - if (static_branch_likely(&mds_user_clear)) - mds_clear_cpu_buffers(); -} - /** * mds_idle_clear_cpu_buffers - Mitigation for MDS vulnerability * diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 10499bcd4e39..75fc0a70877f 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -252,7 +252,7 @@ static void __init mds_select_mitigation(void) if (!boot_cpu_has(X86_FEATURE_MD_CLEAR)) mds_mitigation = MDS_MITIGATION_VMWERV; - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_USER_CLEAR_CPU_BUF); if (!boot_cpu_has(X86_BUG_MSBDS_ONLY) && (mds_nosmt || cpu_mitigations_auto_nosmt())) @@ -356,7 +356,7 @@ static void __init taa_select_mitigation(void) * For guests that can't determine whether the correct microcode is * present on host, enable the mitigation for UCODE_NEEDED as well. */ - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_USER_CLEAR_CPU_BUF); if (taa_nosmt || cpu_mitigations_auto_nosmt()) cpu_smt_disable(false); @@ -424,7 +424,7 @@ static void __init mmio_select_mitigation(void) */ if (boot_cpu_has_bug(X86_BUG_MDS) || (boot_cpu_has_bug(X86_BUG_TAA) && boot_cpu_has(X86_FEATURE_RTM))) - static_branch_enable(&mds_user_clear); + setup_force_cpu_cap(X86_FEATURE_USER_CLEAR_CPU_BUF); else static_branch_enable(&mmio_stale_data_clear); @@ -484,11 +484,11 @@ static void __init md_clear_update_mitigation(void) if (cpu_mitigations_off()) return; - if (!static_key_enabled(&mds_user_clear)) + if (!boot_cpu_has(X86_FEATURE_USER_CLEAR_CPU_BUF)) goto out; /* - * mds_user_clear is now enabled. Update MDS, TAA and MMIO Stale Data + * X86_FEATURE_USER_CLEAR_CPU_BUF is now enabled. Update MDS, TAA and MMIO Stale Data * mitigation, if necessary. */ if (mds_mitigation == MDS_MITIGATION_OFF && diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index a0c551846b35..ebfff8dca661 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -551,8 +551,6 @@ DEFINE_IDTENTRY_RAW(exc_nmi) if (this_cpu_dec_return(nmi_state)) goto nmi_restart; - if (user_mode(regs)) - mds_user_clear_cpu_buffers(); if (IS_ENABLED(CONFIG_NMI_CHECK_CPU)) { WRITE_ONCE(nsp->idt_seq, nsp->idt_seq + 1); WARN_ON_ONCE(nsp->idt_seq & 0x1); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 72e3943f3693..c16297a49e4d 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7229,7 +7229,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, /* L1D Flush includes CPU buffer clear to mitigate MDS */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (static_branch_unlikely(&mds_user_clear)) + else if (cpu_feature_enabled(X86_FEATURE_USER_CLEAR_CPU_BUF)) mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) From patchwork Fri Oct 20 20:45:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 13431109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1545FC0032E for ; Fri, 20 Oct 2023 20:45:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231786AbjJTUpt (ORCPT ); Fri, 20 Oct 2023 16:45:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231792AbjJTUpf (ORCPT ); Fri, 20 Oct 2023 16:45:35 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDFEF1703; Fri, 20 Oct 2023 13:45:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834726; x=1729370726; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=MYGTJ5cvnXWg6uwlltOdXM4ZAEsPGA4t+1Vpxa3NQlw=; b=DYdLoAt55s+cLosXIgh1ArQH+Bz73e2aIAgqNFFYlcz7uMpCMGb2WDaJ Bqgn9zPbX9GZZAp+J3ge3DIa4Lgleem+ZA0BKZXs35KPBWmY1a/a7jKBH ODBEUqQKL2ddwyFbCPCHf/y2K3m/0vW4rJpVBDUBSERW6sGj62SfJULwZ of4BP6YD28YaFQcA+M/Ru5hRTn/4NcBFZDfzQlNJJ6mYVMFSFLJBbCHXC HoAnYNbz0gWrERcUcZ21wY6IR1FXQLVFWZ7+dUkx45cyM9AwKsatG6j8h fQR8GpL6PqQm4PVsDjqsMEPTqEtSFurcyuZ0nHzudJac9KtpAqWI5mLRD A==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="365909429" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="365909429" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:24 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="848201758" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="848201758" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:23 -0700 Date: Fri, 20 Oct 2023 13:45:22 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH 5/6] x86/bugs: Cleanup mds_user_clear Message-ID: <20231020-delay-verw-v1-5-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org There are no more users of mds_user_clear static key, remove it. Signed-off-by: Pawan Gupta --- arch/x86/include/asm/nospec-branch.h | 1 - arch/x86/kernel/cpu/bugs.c | 3 --- 2 files changed, 4 deletions(-) diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index 501c26ecd722..520689325014 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -563,7 +563,6 @@ DECLARE_STATIC_KEY_FALSE(switch_to_cond_stibp); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); DECLARE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -DECLARE_STATIC_KEY_FALSE(mds_user_clear); DECLARE_STATIC_KEY_FALSE(mds_idle_clear); DECLARE_STATIC_KEY_FALSE(switch_mm_cond_l1d_flush); diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 75fc0a70877f..cf2181b45055 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -111,9 +111,6 @@ DEFINE_STATIC_KEY_FALSE(switch_mm_cond_ibpb); /* Control unconditional IBPB in switch_mm() */ DEFINE_STATIC_KEY_FALSE(switch_mm_always_ibpb); -/* Control MDS CPU buffer clear before returning to user space */ -DEFINE_STATIC_KEY_FALSE(mds_user_clear); -EXPORT_SYMBOL_GPL(mds_user_clear); /* Control MDS CPU buffer clear before idling (halt, mwait) */ DEFINE_STATIC_KEY_FALSE(mds_idle_clear); EXPORT_SYMBOL_GPL(mds_idle_clear); From patchwork Fri Oct 20 20:45:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pawan Gupta X-Patchwork-Id: 13431110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B481C001DF for ; Fri, 20 Oct 2023 20:45:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231228AbjJTUp5 (ORCPT ); Fri, 20 Oct 2023 16:45:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231160AbjJTUpp (ORCPT ); Fri, 20 Oct 2023 16:45:45 -0400 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 264701717; Fri, 20 Oct 2023 13:45:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697834732; x=1729370732; h=date:from:to:cc:subject:message-id:references: mime-version:in-reply-to; bh=efcm4OHP2Uz572Un+YAEz/A+CIucgALlZ4w/Rew7Rqg=; b=F/rxuNemKnaWdJ9lk7HdzRn0wZJOjnk0pEjMJE2qTWjQYYbfxpO9ni6z stLc0qUV/lcjFfZubfAZirqtcbolK1gCJ8aVvhfqHc+A5z1QzGruG6RFr B7HWY9JJSzKbDI36O7KjjwaFlg7zp5X6IFNnks6c24/xD9z34v6HrdH6R FK01NAw09dGSJp1dEZut39X7CGRJDRdMaDc4v4YzoOM9XqrNEOUSYqshw ZECUt0iyK9UUuLd2E0WsRiSziaKCYsdIqdZpOlUVuybAkmVETWrUetOR8 ZxnpYG0JqA57GH6bnBV5i4cvBDdMpQ0KKot8/tyjr3OgyP/uzKWVJSgVl w==; X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="365909446" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="365909446" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10869"; a="848201802" X-IronPort-AV: E=Sophos;i="6.03,239,1694761200"; d="scan'208";a="848201802" Received: from hkchanda-mobl.amr.corp.intel.com (HELO desk) ([10.209.90.113]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Oct 2023 13:45:30 -0700 Date: Fri, 20 Oct 2023 13:45:29 -0700 From: Pawan Gupta To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Peter Zijlstra , Josh Poimboeuf , Andy Lutomirski , Jonathan Corbet , Sean Christopherson , Paolo Bonzini , tony.luck@intel.com, ak@linux.intel.com, tim.c.chen@linux.intel.com Cc: linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, Alyssa Milburn , Daniel Sneddon , antonio.gomez.iglesias@linux.intel.com, Pawan Gupta Subject: [PATCH 6/6] KVM: VMX: Move VERW closer to VMentry for MDS mitigation Message-ID: <20231020-delay-verw-v1-6-cff54096326d@linux.intel.com> X-Mailer: b4 0.12.3 References: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20231020-delay-verw-v1-0-cff54096326d@linux.intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org During VMentry VERW is executed to mitigate MDS. After VERW, any memory access like register push onto stack may put host data in MDS affected CPU buffers. A guest can then use MDS to sample host data. Although likelihood of secrets surviving in registers at current VERW callsite is less, but it can't be ruled out. Harden the MDS mitigation by moving the VERW mitigation late in VMentry path. Note that VERW for MMIO Stale Data mitigation is unchanged because of the complexity of per-guest conditional VERW which is not easy to handle that late in asm with no GPRs available. If the CPU is also affected by MDS, VERW is unconditionally executed late in asm regardless of guest having MMIO access. Signed-off-by: Pawan Gupta Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/vmenter.S | 9 +++++++++ arch/x86/kvm/vmx/vmx.c | 10 +++++++--- 2 files changed, 16 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index be275a0410a8..efa716cf4727 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -1,6 +1,7 @@ /* SPDX-License-Identifier: GPL-2.0 */ #include #include +#include #include #include #include @@ -31,6 +32,8 @@ #define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE #endif +#define GUEST_CLEAR_CPU_BUFFERS USER_CLEAR_CPU_BUFFERS + .macro VMX_DO_EVENT_IRQOFF call_insn call_target /* * Unconditionally create a stack frame, getting the correct RSP on the @@ -177,10 +180,16 @@ SYM_FUNC_START(__vmx_vcpu_run) * the 'vmx_vmexit' label below. */ .Lvmresume: + /* Mitigate CPU data sampling attacks .e.g. MDS */ + GUEST_CLEAR_CPU_BUFFERS + vmresume jmp .Lvmfail .Lvmlaunch: + /* Mitigate CPU data sampling attacks .e.g. MDS */ + GUEST_CLEAR_CPU_BUFFERS + vmlaunch jmp .Lvmfail diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c16297a49e4d..e3d0eda292c3 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7226,13 +7226,17 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, guest_state_enter_irqoff(); - /* L1D Flush includes CPU buffer clear to mitigate MDS */ + /* + * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW + * mitigation for MDS is done late in VMentry and is still executed + * inspite of L1D Flush. This is because an extra VERW should not matter + * much after the big hammer L1D Flush. + */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); - else if (cpu_feature_enabled(X86_FEATURE_USER_CLEAR_CPU_BUF)) - mds_clear_cpu_buffers(); else if (static_branch_unlikely(&mmio_stale_data_clear) && kvm_arch_has_assigned_device(vcpu->kvm)) + /* MMIO mitigation is mutually exclusive to MDS mitigation later in asm */ mds_clear_cpu_buffers(); vmx_disable_fb_clear(vmx);