From patchwork Thu Jun 16 08:46:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 957DEC433EF for ; Thu, 16 Jun 2022 08:49:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376355AbiFPItd (ORCPT ); Thu, 16 Jun 2022 04:49:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35618 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376283AbiFPIsr (ORCPT ); Thu, 16 Jun 2022 04:48:47 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27CDD15705; Thu, 16 Jun 2022 01:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369262; x=1686905262; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IKjbYcydGhUQOtO5VnHy7aAu+jLJsu2LqXV0XPrI8+M=; b=UhUIz0bj+7pY7qY4tlLJlE0hP5ByrHMA//lR3Kg3Ty8zBbQzouplfuLc M6FLBZ2r4/Q86hy3JUq+TW34W1dfN+b+uRm6wuv6EC9FUqZNYBHYgwmi4 WjWp5G9zqoXZIl3dVUN5p6GovVanBcO2JLfMejx31W0XlSTHvCUBGW0X9 bJdRsp5wnzT8K3hKAYyyeBUU9Zm2GWZU3I+LzGi2qQ7rbUwggxZZqqzf3 NzXBpNnZ1ZkbqgQgSv4bNAEG6KgHNbMAp992+IjWygSddr7HOwq1Hx2NK 2ZVfERhB6gyJstaC7wolTVVkAKKE0gcAfJYx5ARTEbwRfxvmfoefqyFNB Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664550" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664550" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083128" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Yu-cheng Yu , Kees Cook Subject: [PATCH 01/19] x86/cet/shstk: Add Kconfig option for Shadow Stack Date: Thu, 16 Jun 2022 04:46:25 -0400 Message-Id: <20220616084643.19564-2-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Yu-cheng Yu Shadow Stack provides protection against function return address corruption. It is active when the processor supports it, the kernel has CONFIG_X86_SHADOW_STACK enabled, and the application is built for the feature. This is only implemented for the 64-bit kernel. When it is enabled, legacy non-Shadow Stack applications continue to work, but without protection. Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Signed-off-by: Yang Weijiang Cc: Kees Cook --- v2: - Remove already wrong kernel size increase info (tlgx) - Change prompt to remove "Intel" (tglx) - Update line about what CPUs are supported (Dave) Yu-cheng v25: - Remove X86_CET and use X86_SHADOW_STACK directly. Yu-cheng v24: - Update for the splitting X86_CET to X86_SHADOW_STACK and X86_IBT. arch/x86/Kconfig | 17 +++++++++++++++++ arch/x86/Kconfig.assembler | 1 + 2 files changed, 18 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 9783ebc4e021..79c6b0490350 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -26,6 +26,7 @@ config X86_64 depends on 64BIT # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE + select ARCH_HAS_SHADOW_STACK select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY @@ -1969,6 +1970,22 @@ config X86_SGX If unsure, say N. +config ARCH_HAS_SHADOW_STACK + def_bool n + +config X86_SHADOW_STACK + prompt "X86 Shadow Stack" + def_bool n + depends on ARCH_HAS_SHADOW_STACK + help + Shadow Stack protection is a hardware feature that detects function + return address corruption. Today the kernel's support is limited to + virtualizing it in KVM guests. + + CPUs supporting shadow stacks were first released in 2020. + + If unsure, say N. + config EFI bool "EFI runtime service support" depends on ACPI diff --git a/arch/x86/Kconfig.assembler b/arch/x86/Kconfig.assembler index 26b8c08e2fc4..41428391e475 100644 --- a/arch/x86/Kconfig.assembler +++ b/arch/x86/Kconfig.assembler @@ -19,3 +19,4 @@ config AS_TPAUSE def_bool $(as-instr,tpause %ecx) help Supported by binutils >= 2.31.1 and LLVM integrated assembler >= V7 + From patchwork Thu Jun 16 08:46:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F324C433EF for ; Thu, 16 Jun 2022 08:49:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376470AbiFPItE (ORCPT ); Thu, 16 Jun 2022 04:49:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376304AbiFPIsv (ORCPT ); Thu, 16 Jun 2022 04:48:51 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3FF21901F; Thu, 16 Jun 2022 01:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369263; x=1686905263; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IFsbLhw2T9ri7FM7lx8baIjLwWZ/8GijXSEdM4I3dCE=; b=gRVPUvUh1/FRqq5kxfy62uPr69CMaippso4Py4L++BiP0PVAx94F/GGZ wjUH48yQmKVCb8w/6A1zAFNphREb2OU3QPJjmYXM0jJiewQFJpFPIQRfV 8d3EVsKlbedWSLNnnl5gVhfurfXcB8Q92Of1xM7wgoTtEV4gjRyniso8s NPKy8s8fjaYki5ZJ1a4dLHnnFf78hgxnTft2HrcI60N7pEJ9WbcT+ghvy Ck/e6Q+4HLY0S1hlsD3VaL/b+h+hFrZ94AZMv0l1LTiLQqFKpTokcIZr1 8ikPInhkPcV1lb6ljH9l8h2Ef80TeWdnz0EuDowV6CyK0j0MpsdukCyhg Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664552" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664552" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083131" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Yu-cheng Yu , Kees Cook Subject: [PATCH 02/19] x86/cpufeatures: Add CPU feature flags for shadow stacks Date: Thu, 16 Jun 2022 04:46:26 -0400 Message-Id: <20220616084643.19564-3-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Yu-cheng Yu The Control-Flow Enforcement Technology contains two related features, one of which is Shadow Stacks. Future patches will utilize this feature for shadow stack support in KVM, so add a CPU feature flags for Shadow Stacks (CPUID.(EAX=7,ECX=0):ECX[bit 7]). To protect shadow stack state from malicious modification, the registers are only accessible in supervisor mode. This implementation context-switches the registers with XSAVES. Make X86_FEATURE_SHSTK depend on XSAVES. Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Signed-off-by: Yang Weijiang Cc: Kees Cook --- v2: - Remove IBT reference in commit log (Kees) - Describe xsaves dependency using text from (Dave) v1: - Remove IBT, can be added in a follow on IBT series. Yu-cheng v25: - Make X86_FEATURE_IBT depend on X86_FEATURE_SHSTK. Yu-cheng v24: - Update for splitting CONFIG_X86_CET to CONFIG_X86_SHADOW_STACK and CONFIG_X86_IBT. - Move DISABLE_IBT definition to the IBT series. arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/disabled-features.h | 8 +++++++- arch/x86/kernel/cpu/cpuid-deps.c | 1 + 3 files changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 393f2bbb5e3a..2a3aaf5e1052 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -355,6 +355,7 @@ #define X86_FEATURE_OSPKE (16*32+ 4) /* OS Protection Keys Enable */ #define X86_FEATURE_WAITPKG (16*32+ 5) /* UMONITOR/UMWAIT/TPAUSE Instructions */ #define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* Additional AVX512 Vector Bit Manipulation Instructions */ +#define X86_FEATURE_SHSTK (16*32+ 7) /* Shadow Stack */ #define X86_FEATURE_GFNI (16*32+ 8) /* Galois Field New Instructions */ #define X86_FEATURE_VAES (16*32+ 9) /* Vector AES */ #define X86_FEATURE_VPCLMULQDQ (16*32+10) /* Carry-Less Multiplication Double Quadword */ diff --git a/arch/x86/include/asm/disabled-features.h b/arch/x86/include/asm/disabled-features.h index 36369e76cc63..c61c65bbc58d 100644 --- a/arch/x86/include/asm/disabled-features.h +++ b/arch/x86/include/asm/disabled-features.h @@ -68,6 +68,12 @@ # define DISABLE_TDX_GUEST (1 << (X86_FEATURE_TDX_GUEST & 31)) #endif +#ifdef CONFIG_X86_SHADOW_STACK +#define DISABLE_SHSTK 0 +#else +#define DISABLE_SHSTK (1 << (X86_FEATURE_SHSTK & 31)) +#endif + /* * Make sure to add features to the correct mask */ @@ -88,7 +94,7 @@ #define DISABLED_MASK14 0 #define DISABLED_MASK15 0 #define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP| \ - DISABLE_ENQCMD) + DISABLE_ENQCMD|DISABLE_SHSTK) #define DISABLED_MASK17 0 #define DISABLED_MASK18 0 #define DISABLED_MASK19 0 diff --git a/arch/x86/kernel/cpu/cpuid-deps.c b/arch/x86/kernel/cpu/cpuid-deps.c index c881bcafba7d..bf1b55a1ba21 100644 --- a/arch/x86/kernel/cpu/cpuid-deps.c +++ b/arch/x86/kernel/cpu/cpuid-deps.c @@ -78,6 +78,7 @@ static const struct cpuid_dep cpuid_deps[] = { { X86_FEATURE_XFD, X86_FEATURE_XSAVES }, { X86_FEATURE_XFD, X86_FEATURE_XGETBV1 }, { X86_FEATURE_AMX_TILE, X86_FEATURE_XFD }, + { X86_FEATURE_SHSTK, X86_FEATURE_XSAVES }, {} }; From patchwork Thu Jun 16 08:46:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883582 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75354C43334 for ; Thu, 16 Jun 2022 08:49:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376329AbiFPIta (ORCPT ); Thu, 16 Jun 2022 04:49:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376319AbiFPIsv (ORCPT ); Thu, 16 Jun 2022 04:48:51 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 627261BE88; Thu, 16 Jun 2022 01:47:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369264; x=1686905264; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=FSN5H84gs5D/Dj5DTDmZALwdCo8U9STbNybYgwFnfzI=; b=hGua9T3ZLslif9K98H+1ZuG6OEUTAVy55qzpkheZN4kKDnvPj8lCDU5O 13xZbkLGscPb9ZZXXziIaHOMKVhkL7Vs5XbIncI9I9LeBQZzQ1JRc58L3 Jaoxo7YOFseoZvE0wrbkuDY1JybMnO+KexI+b3sme4fyIeLieYvdvjP61 QGvTXksdFIjsnhUlIiol6lsTHkkyWuvcttv6jEmJnX4wo2KdOYmv1YCIE MzcY1peqHUZYP/jxDajaGMuO2d83T48a1yp2N9H5Lw8FBeHPqNph/71w9 eCs1Ku0mWOb3RncJUsEqwpaewCYeUI2vhOZdDiLbRVW/Xa3owZNm+lc2g Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664554" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664554" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083133" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Yu-cheng Yu , Kees Cook Subject: [PATCH 03/19] x86/cpufeatures: Enable CET CR4 bit for shadow stack Date: Thu, 16 Jun 2022 04:46:27 -0400 Message-Id: <20220616084643.19564-4-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Yu-cheng Yu Utilizing CET features requires a CR4 bit to be enabled as well as bits to be set in CET MSRs. Setting the CR4 bit does two things: 1. Enables the usage of WRUSS instruction, which the kernel can use to write to userspace shadow stacks. 2. Allows those individual aspects of CET to be enabled later via the MSR. While future patches will allow the MSR values to be saved and restored per task, the CR4 bit will allow for WRUSS to be used regardless of if a tasks CET MSRs have been restored. Kernel IBT already enables the CR4 bit. Modify the logic to enable it for when the kernel is configured with and detects shadow stack support, as well. Rename cet_disable() to ibt_disable() since it no longer applies to all CET features in the kernel. Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Signed-off-by: Yang Weijiang Cc: Kees Cook --- v2: - Drop no_user_shstk (Dave Hansen) - Elaborate on what the CR4 bit does in the commit log - Integrate with Kernel IBT logic v1: - Moved kernel-parameters.txt changes here from patch 1. Yu-cheng v25: - Remove software-defined X86_FEATURE_CET. Yu-cheng v24: - Update #ifdef placement to reflect Kconfig changes of splitting shadow stack and ibt. arch/x86/include/asm/cpu.h | 2 +- arch/x86/kernel/cpu/common.c | 14 +++++++------- arch/x86/kernel/machine_kexec_64.c | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/cpu.h b/arch/x86/include/asm/cpu.h index 8cbf623f0ecf..a56270838435 100644 --- a/arch/x86/include/asm/cpu.h +++ b/arch/x86/include/asm/cpu.h @@ -74,7 +74,7 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c); static inline void init_ia32_feat_ctl(struct cpuinfo_x86 *c) {} #endif -extern __noendbr void cet_disable(void); +extern __noendbr void ibt_disable(void); struct ucode_cpu_info; diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c index c296cb1c0113..86102a8d451e 100644 --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -598,23 +598,23 @@ __noendbr void ibt_restore(u64 save) static __always_inline void setup_cet(struct cpuinfo_x86 *c) { + bool kernel_ibt = HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT); u64 msr = CET_ENDBR_EN; - if (!HAS_KERNEL_IBT || - !cpu_feature_enabled(X86_FEATURE_IBT)) - return; + if (kernel_ibt) + wrmsrl(MSR_IA32_S_CET, msr); - wrmsrl(MSR_IA32_S_CET, msr); - cr4_set_bits(X86_CR4_CET); + if (kernel_ibt || cpu_feature_enabled(X86_FEATURE_SHSTK)) + cr4_set_bits(X86_CR4_CET); - if (!ibt_selftest()) { + if (kernel_ibt && !ibt_selftest()) { pr_err("IBT selftest: Failed!\n"); setup_clear_cpu_cap(X86_FEATURE_IBT); return; } } -__noendbr void cet_disable(void) +__noendbr void ibt_disable(void) { if (cpu_feature_enabled(X86_FEATURE_IBT)) wrmsrl(MSR_IA32_S_CET, 0); diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c index 0611fd83858e..745024654fcd 100644 --- a/arch/x86/kernel/machine_kexec_64.c +++ b/arch/x86/kernel/machine_kexec_64.c @@ -311,7 +311,7 @@ void machine_kexec(struct kimage *image) /* Interrupts aren't acceptable while we reboot */ local_irq_disable(); hw_breakpoint_disable(); - cet_disable(); + ibt_disable(); if (image->preserve_context) { #ifdef CONFIG_X86_IO_APIC From patchwork Thu Jun 16 08:46:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02561CCA47A for ; Thu, 16 Jun 2022 08:49:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376380AbiFPItf (ORCPT ); Thu, 16 Jun 2022 04:49:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35324 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376391AbiFPIsy (ORCPT ); Thu, 16 Jun 2022 04:48:54 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B56BE53700; Thu, 16 Jun 2022 01:47:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369267; x=1686905267; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DNQHbD5a674vCUo1VWtLPFj7GhV8y9chgvxSiYSfVNY=; b=IlSDl792FFu4QfRYMCsjgRW/9PNh9LROzOz1OTUQcg1bnvyuEmqznlLx f+e5HcBFGQUB7yCn49VjXNKK8JgkD1Zipo/hxJAcmmosB8feTWBvzkNbU V9tDuYyT2Gkjq1ygc3rJwE5dSemDUbhxIh7ueTW03qEIZRGok5fE+52jg Sq+cQSZu1Kw/64OjwmX1TVG71sBshrduM47uloxphWDitutLc+zALq3Sp upUp2Xgr9DPelbfHuXfw+zpZ2UV/sMkIs9vaq9R+MH4r+N+BCwkfQ9y6N vwIbcMXNjpVxELRJqfz3ZrIo9yV2yA5KsoeqtUa+X21cGH9qcHf1wxvkQ Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664556" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664556" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083137" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Yu-cheng Yu , Kees Cook Subject: [PATCH 04/19] x86/fpu/xstate: Introduce CET MSR and XSAVES supervisor states Date: Thu, 16 Jun 2022 04:46:28 -0400 Message-Id: <20220616084643.19564-5-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Yu-cheng Yu Shadow stack register state can be managed with XSAVE. The registers can logically be separated into two groups: * Registers controlling user-mode operation * Registers controlling kernel-mode operation The architecture has two new XSAVE state components: one for each group of those groups of registers. This lets an OS manage them separately if it chooses. Future patches for host userspace and KVM guests will only utilize the user-mode registers, so only configure XSAVE to save user-mode registers. This state will add 16 bytes to the xsave buffer size. Future patches will use the user-mode XSAVE area to save guest user-mode CET state. However, VMCS includes new fields for guest CET supervisor states. KVM can use these to save and restore guest supervisor state, so host supervisor XSAVE support is not required. Adding this exacerbates the already unwieldy if statement in check_xstate_against_struct() that handles warning about un-implemented xfeatures. So refactor these check's by having XCHECK_SZ() set a bool when it actually check's the xfeature. This ends up exceeding 80 chars, but was better on balance than other options explored. Pass the bool as pointer to make it clear that XCHECK_SZ() can change the variable. While configuring user-mode XSAVE, clarify kernel-mode registers are not managed by XSAVE by defining the xfeature in XFEATURE_MASK_SUPERVISOR_UNSUPPORTED, like is done for XFEATURE_MASK_PT. This serves more of a documentation as code purpose, and functionally, only enables a few safety checks. Both XSAVE state components are supervisor states, even the state controlling user-mode operation. This is a departure from earlier features like protection keys where the PKRU state a normal user (non-supervisor) state. Having the user state be supervisor-managed ensures there is no direct, unprivileged access to it, making it harder for an attacker to subvert CET. To facilitate this privileged access, define the two user-mode CET MSRs, and the bits defined in those MSRs relevant to future shadow stack enablement patches. Signed-off-by: Yu-cheng Yu Co-developed-by: Rick Edgecombe Signed-off-by: Rick Edgecombe Signed-off-by: Yang Weijiang Cc: Kees Cook --- v2: - Reword commit log using some verbiage posted by Dave Hansen - Remove unlikely to be used supervisor cet xsave struct - Clarify that supervisor cet state is not saved by xsave - Remove unused supervisor MSRs v1: - Remove outdated reference to sigreturn checks on msr's. Yu-cheng v29: - Move CET MSR definition up in msr-index.h. Yu-cheng v28: - Add XFEATURE_MASK_CET_USER to XFEATURES_INIT_FPSTATE_HANDLED. Yu-cheng v25: - Update xsave_cpuid_features[]. Now CET XSAVES features depend on X86_FEATURE_SHSTK (vs. the software-defined X86_FEATURE_CET). arch/x86/include/asm/fpu/types.h | 14 ++++- arch/x86/include/asm/fpu/xstate.h | 6 +- arch/x86/kernel/fpu/xstate.c | 93 ++++++++++++++++--------------- 3 files changed, 63 insertions(+), 50 deletions(-) diff --git a/arch/x86/include/asm/fpu/types.h b/arch/x86/include/asm/fpu/types.h index eb7cd1139d97..03aa98fb9c2b 100644 --- a/arch/x86/include/asm/fpu/types.h +++ b/arch/x86/include/asm/fpu/types.h @@ -115,8 +115,8 @@ enum xfeature { XFEATURE_PT_UNIMPLEMENTED_SO_FAR, XFEATURE_PKRU, XFEATURE_PASID, - XFEATURE_RSRVD_COMP_11, - XFEATURE_RSRVD_COMP_12, + XFEATURE_CET_USER, + XFEATURE_CET_KERNEL_UNIMPLEMENTED_SO_FAR, XFEATURE_RSRVD_COMP_13, XFEATURE_RSRVD_COMP_14, XFEATURE_LBR, @@ -138,6 +138,8 @@ enum xfeature { #define XFEATURE_MASK_PT (1 << XFEATURE_PT_UNIMPLEMENTED_SO_FAR) #define XFEATURE_MASK_PKRU (1 << XFEATURE_PKRU) #define XFEATURE_MASK_PASID (1 << XFEATURE_PASID) +#define XFEATURE_MASK_CET_USER (1 << XFEATURE_CET_USER) +#define XFEATURE_MASK_CET_KERNEL (1 << XFEATURE_CET_KERNEL_UNIMPLEMENTED_SO_FAR) #define XFEATURE_MASK_LBR (1 << XFEATURE_LBR) #define XFEATURE_MASK_XTILE_CFG (1 << XFEATURE_XTILE_CFG) #define XFEATURE_MASK_XTILE_DATA (1 << XFEATURE_XTILE_DATA) @@ -252,6 +254,14 @@ struct pkru_state { u32 pad; } __packed; +/* + * State component 11 is Control-flow Enforcement user states + */ +struct cet_user_state { + u64 user_cet; /* user control-flow settings */ + u64 user_ssp; /* user shadow stack pointer */ +}; + /* * State component 15: Architectural LBR configuration state. * The size of Arch LBR state depends on the number of LBRs (lbr_depth). diff --git a/arch/x86/include/asm/fpu/xstate.h b/arch/x86/include/asm/fpu/xstate.h index cd3dd170e23a..d4427b88ee12 100644 --- a/arch/x86/include/asm/fpu/xstate.h +++ b/arch/x86/include/asm/fpu/xstate.h @@ -50,7 +50,8 @@ #define XFEATURE_MASK_USER_DYNAMIC XFEATURE_MASK_XTILE_DATA /* All currently supported supervisor features */ -#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID) +#define XFEATURE_MASK_SUPERVISOR_SUPPORTED (XFEATURE_MASK_PASID | \ + XFEATURE_MASK_CET_USER) /* * A supervisor state component may not always contain valuable information, @@ -77,7 +78,8 @@ * Unsupported supervisor features. When a supervisor feature in this mask is * supported in the future, move it to the supported supervisor feature mask. */ -#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT) +#define XFEATURE_MASK_SUPERVISOR_UNSUPPORTED (XFEATURE_MASK_PT | \ + XFEATURE_MASK_CET_KERNEL) /* All supervisor states including supported and unsupported states. */ #define XFEATURE_MASK_SUPERVISOR_ALL (XFEATURE_MASK_SUPERVISOR_SUPPORTED | \ diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c index c8340156bfd2..5e6a4867fd05 100644 --- a/arch/x86/kernel/fpu/xstate.c +++ b/arch/x86/kernel/fpu/xstate.c @@ -39,26 +39,26 @@ */ static const char *xfeature_names[] = { - "x87 floating point registers" , - "SSE registers" , - "AVX registers" , - "MPX bounds registers" , - "MPX CSR" , - "AVX-512 opmask" , - "AVX-512 Hi256" , - "AVX-512 ZMM_Hi256" , - "Processor Trace (unused)" , - "Protection Keys User registers", - "PASID state", - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "unknown xstate feature" , - "AMX Tile config" , - "AMX Tile data" , - "unknown xstate feature" , + "x87 floating point registers" , + "SSE registers" , + "AVX registers" , + "MPX bounds registers" , + "MPX CSR" , + "AVX-512 opmask" , + "AVX-512 Hi256" , + "AVX-512 ZMM_Hi256" , + "Processor Trace (unused)" , + "Protection Keys User registers" , + "PASID state" , + "Control-flow User registers" , + "Control-flow Kernel registers (unused)" , + "unknown xstate feature" , + "unknown xstate feature" , + "unknown xstate feature" , + "unknown xstate feature" , + "AMX Tile config" , + "AMX Tile data" , + "unknown xstate feature" , }; static unsigned short xsave_cpuid_features[] __initdata = { @@ -73,6 +73,7 @@ static unsigned short xsave_cpuid_features[] __initdata = { [XFEATURE_PT_UNIMPLEMENTED_SO_FAR] = X86_FEATURE_INTEL_PT, [XFEATURE_PKRU] = X86_FEATURE_PKU, [XFEATURE_PASID] = X86_FEATURE_ENQCMD, + [XFEATURE_CET_USER] = X86_FEATURE_SHSTK, [XFEATURE_XTILE_CFG] = X86_FEATURE_AMX_TILE, [XFEATURE_XTILE_DATA] = X86_FEATURE_AMX_TILE, }; @@ -276,6 +277,7 @@ static void __init print_xstate_features(void) print_xstate_feature(XFEATURE_MASK_Hi16_ZMM); print_xstate_feature(XFEATURE_MASK_PKRU); print_xstate_feature(XFEATURE_MASK_PASID); + print_xstate_feature(XFEATURE_MASK_CET_USER); print_xstate_feature(XFEATURE_MASK_XTILE_CFG); print_xstate_feature(XFEATURE_MASK_XTILE_DATA); } @@ -344,6 +346,7 @@ static __init void os_xrstor_booting(struct xregs_state *xstate) XFEATURE_MASK_BNDREGS | \ XFEATURE_MASK_BNDCSR | \ XFEATURE_MASK_PASID | \ + XFEATURE_MASK_CET_USER | \ XFEATURE_MASK_XTILE) /* @@ -446,13 +449,14 @@ static void __init __xstate_dump_leaves(void) } \ } while (0) -#define XCHECK_SZ(sz, nr, nr_macro, __struct) do { \ - if ((nr == nr_macro) && \ - WARN_ONCE(sz != sizeof(__struct), \ - "%s: struct is %zu bytes, cpu state %d bytes\n", \ - __stringify(nr_macro), sizeof(__struct), sz)) { \ - __xstate_dump_leaves(); \ - } \ +#define XCHECK_SZ(checked, sz, nr, nr_macro, __struct) do { \ + if (nr == nr_macro) { \ + *checked = true; \ + if (WARN_ONCE(sz != sizeof(__struct), \ + "%s: struct is %zu bytes, cpu state %d bytes\n", \ + __stringify(nr_macro), sizeof(__struct), sz)) \ + __xstate_dump_leaves(); \ + } \ } while (0) /** @@ -527,33 +531,30 @@ static bool __init check_xstate_against_struct(int nr) * Ask the CPU for the size of the state. */ int sz = xfeature_size(nr); + bool chked = false; + /* * Match each CPU state with the corresponding software * structure. */ - XCHECK_SZ(sz, nr, XFEATURE_YMM, struct ymmh_struct); - XCHECK_SZ(sz, nr, XFEATURE_BNDREGS, struct mpx_bndreg_state); - XCHECK_SZ(sz, nr, XFEATURE_BNDCSR, struct mpx_bndcsr_state); - XCHECK_SZ(sz, nr, XFEATURE_OPMASK, struct avx_512_opmask_state); - XCHECK_SZ(sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state); - XCHECK_SZ(sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); - XCHECK_SZ(sz, nr, XFEATURE_PKRU, struct pkru_state); - XCHECK_SZ(sz, nr, XFEATURE_PASID, struct ia32_pasid_state); - XCHECK_SZ(sz, nr, XFEATURE_XTILE_CFG, struct xtile_cfg); + XCHECK_SZ(&chked, sz, nr, XFEATURE_YMM, struct ymmh_struct); + XCHECK_SZ(&chked, sz, nr, XFEATURE_BNDREGS, struct mpx_bndreg_state); + XCHECK_SZ(&chked, sz, nr, XFEATURE_BNDCSR, struct mpx_bndcsr_state); + XCHECK_SZ(&chked, sz, nr, XFEATURE_OPMASK, struct avx_512_opmask_state); + XCHECK_SZ(&chked, sz, nr, XFEATURE_ZMM_Hi256, struct avx_512_zmm_uppers_state); + XCHECK_SZ(&chked, sz, nr, XFEATURE_Hi16_ZMM, struct avx_512_hi16_state); + XCHECK_SZ(&chked, sz, nr, XFEATURE_PKRU, struct pkru_state); + XCHECK_SZ(&chked, sz, nr, XFEATURE_PASID, struct ia32_pasid_state); + XCHECK_SZ(&chked, sz, nr, XFEATURE_XTILE_CFG, struct xtile_cfg); + XCHECK_SZ(&chked, sz, nr, XFEATURE_CET_USER, struct cet_user_state); /* The tile data size varies between implementations. */ - if (nr == XFEATURE_XTILE_DATA) + if (nr == XFEATURE_XTILE_DATA) { check_xtile_data_against_struct(sz); + chked = true; + } - /* - * Make *SURE* to add any feature numbers in below if - * there are "holes" in the xsave state component - * numbers. - */ - if ((nr < XFEATURE_YMM) || - (nr >= XFEATURE_MAX) || - (nr == XFEATURE_PT_UNIMPLEMENTED_SO_FAR) || - ((nr >= XFEATURE_RSRVD_COMP_11) && (nr <= XFEATURE_RSRVD_COMP_16))) { + if (!chked) { WARN_ONCE(1, "no structure for xstate: %d\n", nr); XSTATE_WARN_ON(1); return false; From patchwork Thu Jun 16 08:46:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB4D9C433EF for ; Thu, 16 Jun 2022 08:49:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359672AbiFPItP (ORCPT ); Thu, 16 Jun 2022 04:49:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1359798AbiFPIs5 (ORCPT ); Thu, 16 Jun 2022 04:48:57 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7966D424B3; Thu, 16 Jun 2022 01:47:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369272; x=1686905272; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=GeWyVkmGWKHPsJgw/B+NoLBgTSesUAxxZOkxfdZhD8M=; b=M9Lb0UZYW77ejhpzJ3WIAy0cr79DmW+jn8PdKyDn+RLjC/9JSFkslhA6 Fy8gDw3wK0UnEzUhhhKv+1s/wb0Xb/zMZiyo31QUeepsLM0zp9SQcZOAZ w9oSHaPkflT9Ae5LYhWoJd1MP6aWja5lZQ5v9id7a2ptU1hbxRL0HZsBb 01NCGX8PH4U1VG6m8WZ34TwDfR6LBddo7FoqDXCJJ2wkZbkrJOkJ4J4p/ gK+Eg2XQBkRUB3hiKW6GViREKBcHVf6XW4gyVCFdn3fvUcvEHUXrUFr+6 lolK3EDXFrUNNPfLpHfjUSS3wLwv10a2maX29W5wXWP+V+J4KcmogLLPj g==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664559" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664559" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083140" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Thomas Gleixner Subject: [PATCH 05/19] x86/fpu: Add helper for modifying xstate Date: Thu, 16 Jun 2022 04:46:29 -0400 Message-Id: <20220616084643.19564-6-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Rick Edgecombe Just like user xfeatures, supervisor xfeatures can be active in the registers or present in the task FPU buffer. If the registers are active, the registers can be modified directly. If the registers are not active, the modification must be performed on the task FPU buffer. When the state is not active, the kernel could perform modifications directly to the buffer. But in order for it to do that, it needs to know where in the buffer the specific state it wants to modify is located. Doing this is not robust against optimizations that compact the FPU buffer, as each access would require computing where in the buffer it is. The easiest way to modify supervisor xfeature data is to force restore the registers and write directly to the MSRs. Often times this is just fine anyway as the registers need to be restored before returning to userspace. Do this for now, leaving buffer writing optimizations for the future. Add a new function fpregs_lock_and_load() that can simultaneously call fpregs_lock() and do this restore. Also perform some extra sanity checks in this function since this will be used in non-fpu focused code. Suggested-by: Thomas Gleixner Signed-off-by: Rick Edgecombe Signed-off-by: Yang Weijiang --- v2: - Drop optimization of writing directly the buffer, and change API accordingly. - fpregs_lock_and_load() suggested by tglx - Some commit log verbiage from dhansen v1: - New patch. arch/x86/include/asm/fpu/api.h | 7 ++++++- arch/x86/kernel/fpu/core.c | 19 +++++++++++++++++++ 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/fpu/api.h b/arch/x86/include/asm/fpu/api.h index 6b0f31fb53f7..4f34812b4dd5 100644 --- a/arch/x86/include/asm/fpu/api.h +++ b/arch/x86/include/asm/fpu/api.h @@ -82,6 +82,12 @@ static inline void fpregs_unlock(void) preempt_enable(); } +/* + * Lock and load the fpu state into the registers, if they are not already + * loaded. + */ +void fpu_lock_and_load(void); + #ifdef CONFIG_X86_DEBUG_FPU extern void fpregs_assert_state_consistent(void); #else @@ -163,5 +169,4 @@ static inline bool fpstate_is_confidential(struct fpu_guest *gfpu) /* prctl */ extern long fpu_xstate_prctl(int option, unsigned long arg2); - #endif /* _ASM_X86_FPU_API_H */ diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c index 0531d6a06df5..4d250dba1619 100644 --- a/arch/x86/kernel/fpu/core.c +++ b/arch/x86/kernel/fpu/core.c @@ -756,6 +756,25 @@ void switch_fpu_return(void) } EXPORT_SYMBOL_GPL(switch_fpu_return); +void fpu_lock_and_load(void) +{ + /* + * fpregs_lock() only disables preemption (mostly). So modifing state + * in an interrupt could screw up some in progress fpregs operation, + * but appear to work. Warn about it. + */ + WARN_ON_ONCE(!irq_fpu_usable()); + WARN_ON_ONCE(current->flags & PF_KTHREAD); + + fpregs_lock(); + + fpregs_assert_state_consistent(); + + if (test_thread_flag(TIF_NEED_FPU_LOAD)) + fpregs_restore_userregs(); +} +EXPORT_SYMBOL_GPL(fpu_lock_and_load); + #ifdef CONFIG_X86_DEBUG_FPU /* * If current FPU state according to its tracking (loaded FPU context on this From patchwork Thu Jun 16 08:46:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8862BCCA47D for ; Thu, 16 Jun 2022 08:49:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376508AbiFPItG (ORCPT ); Thu, 16 Jun 2022 04:49:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376393AbiFPIsy (ORCPT ); Thu, 16 Jun 2022 04:48:54 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 353535DE46; Thu, 16 Jun 2022 01:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369268; x=1686905268; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Bgt6iiOS05UazPFX6/lMSu/x6EY0lL69MxEoOLr0oBE=; b=COcGx2164nFRUgYlVPWWibnEnb7+KUXJxpz9BMsoZZ5EOlxBA1smqpAQ 7IAO8+4wDLmDUDZhHn21NYY10ixf8cLJsLUTZL9wOzRL/ATCU9X0dyFev 1ixBAw+0/QcWe8n/CPUKFWOcmtTNOonMs1kOmV/KhskWISSLyvGIMdW+6 T8DTrWbhHhcIuo6q32S3aGrkqJmdYt1BrdHQkvj5Ir7jk7VfHvsZOQw1q vzRWWFOOtNm/KFx6Or/Hzi7yBLq4891GWMZU2An5z2oOevhN8DM43Xeap qyMBv54lraM6f8vR194xytLcDF/Lf8YWr74KyH2v/xAbEZPv10bxP5nM8 g==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259055229" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259055229" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083143" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Sean Christopherson Subject: [PATCH 06/19] KVM: x86: Report XSS as an MSR to be saved if there are supported features Date: Thu, 16 Jun 2022 04:46:30 -0400 Message-Id: <20220616084643.19564-7-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson Add MSR_IA32_XSS to the list of MSRs reported to userspace if supported_xss is non-zero, i.e. KVM supports at least one XSS based feature. Signed-off-by: Sean Christopherson Signed-off-by: Yang Weijiang Message-Id: <20220517154100.29983-4-weijiang.yang@intel.com> Signed-off-by: Paolo Bonzini --- arch/x86/kvm/x86.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 2318a99139fa..f525228168b8 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1446,6 +1446,7 @@ static const u32 msrs_to_save_all[] = { MSR_F15H_PERF_CTR0, MSR_F15H_PERF_CTR1, MSR_F15H_PERF_CTR2, MSR_F15H_PERF_CTR3, MSR_F15H_PERF_CTR4, MSR_F15H_PERF_CTR5, MSR_IA32_XFD, MSR_IA32_XFD_ERR, + MSR_IA32_XSS, }; static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_all)]; @@ -6780,6 +6781,10 @@ static void kvm_init_msr_list(void) if (!kvm_cpu_cap_has(X86_FEATURE_XFD)) continue; break; + case MSR_IA32_XSS: + if (!kvm_caps.supported_xss) + continue; + break; default: break; } From patchwork Thu Jun 16 08:46:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD581C433EF for ; Thu, 16 Jun 2022 08:49:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376489AbiFPItF (ORCPT ); Thu, 16 Jun 2022 04:49:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376318AbiFPIsv (ORCPT ); Thu, 16 Jun 2022 04:48:51 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC8231A3AC; Thu, 16 Jun 2022 01:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369263; x=1686905263; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IQkHWXIhJkLWuzUiRVSy3/hnrnF2/6up2mxzDX06RXk=; b=g1tyla9ZBRLbEmnA5s4+Lt5aVyRyeTG2xbu2N1/uN7rvlxqtRBuD3Fad HtRmcFFEvSaEW7Lk+HZKpIeu2jyjvok/z+an8TZCY3WsvpyUkkOBMpC7V k4A/kaw17JMZMOjunDLnpuFzW259URG8YnOg21ABP1mpidQ0iww9zYYBt dn7eXt75SEp+kZiv1JbgxHSjJIM9HumIvtVV80tI3Y7tYfyVlhwOAepoU yxvVYrXeCGZQnphVhnH92Nl5mp4LFzHQtnRhK809sEe9Mi8LxDTVHk2km CMOm43atldtYNUa3skmA6sn1oabuW0E4tkkhnp9QcsgRKcFPAcVfOLVwU w==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664553" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664553" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083146" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Zhang Yi Z Subject: [PATCH 07/19] KVM: x86: Refresh CPUID on writes to MSR_IA32_XSS Date: Thu, 16 Jun 2022 04:46:31 -0400 Message-Id: <20220616084643.19564-8-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Updated CPUID.0xD.0x1, which reports the current required storage size of all features enabled via XCR0 | XSS, when the guest's XSS is modified. Note, KVM does not yet support any XSS based features, i.e. supported_xss is guaranteed to be zero at this time. Co-developed-by: Zhang Yi Z Signed-off-by: Zhang Yi Z Signed-off-by: Yang Weijiang Message-Id: <20220517154100.29983-5-weijiang.yang@intel.com> Signed-off-by: Paolo Bonzini --- arch/x86/kvm/cpuid.c | 16 +++++++++++++--- arch/x86/kvm/x86.c | 6 ++++-- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index d47222ab8e6e..46ca0f1abbcb 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -240,9 +240,19 @@ static void __kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu, struct kvm_cpuid_e best->ebx = xstate_required_size(vcpu->arch.xcr0, false); best = cpuid_entry2_find(entries, nent, 0xD, 1); - if (best && (cpuid_entry_has(best, X86_FEATURE_XSAVES) || - cpuid_entry_has(best, X86_FEATURE_XSAVEC))) - best->ebx = xstate_required_size(vcpu->arch.xcr0, true); + if (best) { + if (cpuid_entry_has(best, X86_FEATURE_XSAVES) || + cpuid_entry_has(best, X86_FEATURE_XSAVEC)) { + u64 xstate = vcpu->arch.xcr0 | vcpu->arch.ia32_xss; + + best->ebx = xstate_required_size(xstate, true); + } + + if (!cpuid_entry_has(best, X86_FEATURE_XSAVES)) { + best->ecx = 0; + best->edx = 0; + } + } best = __kvm_find_kvm_cpuid_features(vcpu, entries, nent); if (kvm_hlt_in_guest(vcpu->kvm) && best && diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index f525228168b8..06fbd3daf393 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -3605,8 +3605,10 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) */ if (data & ~kvm_caps.supported_xss) return 1; - vcpu->arch.ia32_xss = data; - kvm_update_cpuid_runtime(vcpu); + if (vcpu->arch.ia32_xss != data) { + vcpu->arch.ia32_xss = data; + kvm_update_cpuid_runtime(vcpu); + } break; case MSR_SMI_COUNT: if (!msr_info->host_initiated) From patchwork Thu Jun 16 08:46:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1273CC43334 for ; Thu, 16 Jun 2022 08:49:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376405AbiFPItN (ORCPT ); Thu, 16 Jun 2022 04:49:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1359805AbiFPIs5 (ORCPT ); Thu, 16 Jun 2022 04:48:57 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF1BE47073; Thu, 16 Jun 2022 01:47:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369272; x=1686905272; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=v/7Hxi1rfyLkWH5fyO+/GGs23bb1kzLn7mLAwHLFcrM=; b=TMplK1Oas0gDOjny6Jx05R1OaoGfDsJw8db9F9a+z8cKs1Eu/GDCY8V5 pwpjU1F8nNgeBM/uaannLKeJw/qHHmHZdGp6hBEtrwV8nuaif3owVX9t/ xgIgJU6e4vPT4m3RlQM5uClhj3NCbkDEJZbECzYFq1Ef8p9w2ZI5oiiTh vTen0JaDlyclqsqtoTZ2W6iX5Qw0mub0G1pmVye7tugEPWa5nkb26n0S/ jOsHYbH7MCRgYo5fNnQ7er0CEhk/8SlEF7OY7q27qSQU255yA9svLCVim l1xVuhwpLZ5F3OF6oGE3c/3rZ86CMyYoNiVRY7DPYb/8f5sXul2l7qvTS A==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664560" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664560" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083149" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Sean Christopherson Subject: [PATCH 08/19] KVM: x86: Load guest fpu state when accessing MSRs managed by XSAVES Date: Thu, 16 Jun 2022 04:46:32 -0400 Message-Id: <20220616084643.19564-9-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Sean Christopherson If new feature MSRs are supported in XSS and passed through to the guest they are saved and restored by XSAVES/XRSTORS, i.e. in the guest's FPU state. Load the guest's FPU state if userspace is accessing MSRs whose values are managed by XSAVES so that the MSR helper, e.g. kvm_{get,set}_xsave_msr(), can simply do {RD,WR}MSR to access the guest's value. Because is also used for the KVM_GET_MSRS device ioctl(), explicitly check that @vcpu is non-null before attempting to load guest state. The XSS supporting MSRs cannot be retrieved via the device ioctl() without loading guest FPU state (which doesn't exist). Note that guest_cpuid_has() is not queried as host userspace is allowed to access MSRs that have not been exposed to the guest, e.g. it might do KVM_SET_MSRS prior to KVM_SET_CPUID2. Signed-off-by: Sean Christopherson Co-developed-by: Yang Weijiang Signed-off-by: Yang Weijiang --- arch/x86/kvm/x86.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 06fbd3daf393..506454e17afc 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -130,6 +130,9 @@ static int kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu); static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); +static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu); +static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu); + struct kvm_x86_ops kvm_x86_ops __read_mostly; #define KVM_X86_OP(func) \ @@ -4138,6 +4141,11 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) } EXPORT_SYMBOL_GPL(kvm_get_msr_common); +static bool is_xsaves_msr(u32 index) +{ + return index == MSR_IA32_U_CET || index == MSR_IA32_PL3_SSP; +} + /* * Read or write a bunch of msrs. All parameters are kernel addresses. * @@ -4148,11 +4156,20 @@ static int __msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs, int (*do_msr)(struct kvm_vcpu *vcpu, unsigned index, u64 *data)) { + bool fpu_loaded = false; int i; - for (i = 0; i < msrs->nmsrs; ++i) + for (i = 0; i < msrs->nmsrs; ++i) { + if (vcpu && !fpu_loaded && kvm_caps.supported_xss && + is_xsaves_msr(entries[i].index)) { + kvm_load_guest_fpu(vcpu); + fpu_loaded = true; + } if (do_msr(vcpu, entries[i].index, &entries[i].data)) break; + } + if (fpu_loaded) + kvm_put_guest_fpu(vcpu); return i; } From patchwork Thu Jun 16 08:46:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCC10C43334 for ; Thu, 16 Jun 2022 08:49:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376538AbiFPItj (ORCPT ); Thu, 16 Jun 2022 04:49:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35318 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376379AbiFPIsx (ORCPT ); Thu, 16 Jun 2022 04:48:53 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FEA54756F; Thu, 16 Jun 2022 01:47:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369267; x=1686905267; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=axZQXpthMwv42bC9pB1QFXId7O4Vr4FbCohL9wqJtaU=; b=PVKk0g00E6Li2JnQ6ZKvZkiYSLZqGCmq7EqcFXCBrLA33bCJieAZatja PhcmulDrqHk6RIJ/KxVdSs8vsLBWrP+Rh5Wc9YzCMXUfJImTfEAkCow6I a6EpN4iAve38vtpMVsHYSxSbKvJcTB6OQLZ8rWfyScUjdllvRsvvzW2QP L3oHPOP2J2dnf5VzCMtvy2cZup/uXVth84fL6KGViidX/ofjD1Bl3FzDl h+y4VLY6iSaXxWcr8bLlQUTx15VObV7nnkdoC7JmDnPdv43UzaqF8Rfl3 PJtjuokr975YNPYcwzmjhhJ/9+vdn4/U4CeegP+uJGfudc/BFg6uQ0f9y g==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664555" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664555" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083152" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com Subject: [PATCH 09/19] KVM: x86: Add #CP support in guest exception classification. Date: Thu, 16 Jun 2022 04:46:33 -0400 Message-Id: <20220616084643.19564-10-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add handling for Control Protection (#CP) exceptions, vector 21, used and introduced by Intel's Control-Flow Enforcement Technology (CET). relevant CET violation case. See Intel's SDM for details. Signed-off-by: Yang Weijiang Message-Id: <20210203113421.5759-5-weijiang.yang@intel.com> Signed-off-by: Paolo Bonzini --- arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/vmx/nested.c | 2 +- arch/x86/kvm/x86.c | 10 +++++++--- arch/x86/kvm/x86.h | 13 ++++++++++--- 4 files changed, 19 insertions(+), 7 deletions(-) diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kvm.h index 50a4e787d5e6..69146e7436af 100644 --- a/arch/x86/include/uapi/asm/kvm.h +++ b/arch/x86/include/uapi/asm/kvm.h @@ -32,6 +32,7 @@ #define MC_VECTOR 18 #define XM_VECTOR 19 #define VE_VECTOR 20 +#define CP_VECTOR 21 /* Select x86 specific features in */ #define __KVM_HAVE_PIT diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 7d8cd0ebcc75..01fe23c6fa49 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -2819,7 +2819,7 @@ static int nested_check_vm_entry_controls(struct kvm_vcpu *vcpu, /* VM-entry interruption-info field: deliver error code */ should_have_error_code = intr_type == INTR_TYPE_HARD_EXCEPTION && prot_mode && - x86_exception_has_error_code(vector); + x86_exception_has_error_code(vcpu, vector); if (CC(has_error_code != should_have_error_code)) return -EINVAL; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 506454e17afc..40749e47cda7 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -511,11 +511,15 @@ EXPORT_SYMBOL_GPL(kvm_spurious_fault); #define EXCPT_CONTRIBUTORY 1 #define EXCPT_PF 2 -static int exception_class(int vector) +static int exception_class(struct kvm_vcpu *vcpu, int vector) { switch (vector) { case PF_VECTOR: return EXCPT_PF; + case CP_VECTOR: + if (vcpu->arch.cr4_guest_rsvd_bits & X86_CR4_CET) + return EXCPT_BENIGN; + return EXCPT_CONTRIBUTORY; case DE_VECTOR: case TS_VECTOR: case NP_VECTOR: @@ -659,8 +663,8 @@ static void kvm_multiple_exception(struct kvm_vcpu *vcpu, kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu); return; } - class1 = exception_class(prev_nr); - class2 = exception_class(nr); + class1 = exception_class(vcpu, prev_nr); + class2 = exception_class(vcpu, nr); if ((class1 == EXCPT_CONTRIBUTORY && class2 == EXCPT_CONTRIBUTORY) || (class1 == EXCPT_PF && class2 != EXCPT_BENIGN)) { /* diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 501b884b8cc4..b9b1fff6d97a 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -148,13 +148,20 @@ static inline bool is_64_bit_hypercall(struct kvm_vcpu *vcpu) return vcpu->arch.guest_state_protected || is_64_bit_mode(vcpu); } -static inline bool x86_exception_has_error_code(unsigned int vector) +static inline bool x86_exception_has_error_code(struct kvm_vcpu *vcpu, + unsigned int vector) { static u32 exception_has_error_code = BIT(DF_VECTOR) | BIT(TS_VECTOR) | BIT(NP_VECTOR) | BIT(SS_VECTOR) | BIT(GP_VECTOR) | - BIT(PF_VECTOR) | BIT(AC_VECTOR); + BIT(PF_VECTOR) | BIT(AC_VECTOR) | BIT(CP_VECTOR); - return (1U << vector) & exception_has_error_code; + if (!((1U << vector) & exception_has_error_code)) + return false; + + if (vector == CP_VECTOR) + return !(vcpu->arch.cr4_guest_rsvd_bits & X86_CR4_CET); + + return true; } static inline bool mmu_is_nested(struct kvm_vcpu *vcpu) From patchwork Thu Jun 16 08:46:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883586 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A237C43334 for ; Thu, 16 Jun 2022 08:49:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376281AbiFPIth (ORCPT ); Thu, 16 Jun 2022 04:49:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35280 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376381AbiFPIsy (ORCPT ); Thu, 16 Jun 2022 04:48:54 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DEBBB55345; Thu, 16 Jun 2022 01:47:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369267; x=1686905267; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JsIh7oKQdnGxj98JCNvo7OifsO3pR9D4QbYf4cQgaAM=; b=JxcGNOVyX7LnV+W5tz7sl5ElcKYE5S2qJFQLpxeziz0Z7ENxOYG8uJmc mnJy3usL8TQ22XZ85b5tRIDfcYVzqGKS9JpCL9xgVuPA1Frw1QMvCJD4C g49sBXBdHll3xRJXsWLdG9g/VRJFNrf1GVlZI60pvEkDjxpk+W3zegmO/ xzJMDCAupVqLIZRZKxomqjrBjLnaDPRSWvhsJ/C2XJQMs9ypY1Y6bpybZ uMg431aoNwNaAv05H3ongF1ChSnWa+Yuo7g3D78yQVm3uskYU5C2xwSmv RcC3oM9atVu0TJ7z/qyzJJOMDRkmz4AFD0hFoFocOcuMjRvxqT8NmYqzB Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664557" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664557" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083155" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Zhang Yi Z Subject: [PATCH 10/19] KVM: VMX: Introduce CET VMCS fields and flags Date: Thu, 16 Jun 2022 04:46:34 -0400 Message-Id: <20220616084643.19564-11-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org CET (Control-flow Enforcement Technology) is a CPU feature used to prevent Return/Jump-Oriented Programming (ROP/JOP) attacks. CET introduces a new exception type, Control Protection (#CP), and two sub-features to defend against ROP/JOP style control-flow subversion attacks: Shadow Stack (SHSTK): A shadow stack is a second stack used exclusively for control transfer operations. The shadow stack is separate from the data/normal stack and can be enabled individually in user and kernel mode. When shadow stacks are enabled, CALL pushes the return address on both the data and shadow stack. RET pops the return address from both stacks and compares them. If the return addresses from the two stacks do not match, the processor signals a #CP. Indirect Branch Tracking (IBT): IBT adds a new instrution, ENDBRANCH, that is used to mark valid target addresses of indirect branches (CALL, JMP, ENCLU[EEXIT], etc...). If an indirect branch is executed and the next instruction is _not_ an ENDBRANCH, the processor signals a #CP. Several new CET MSRs are defined to support CET: MSR_IA32_{U,S}_CET: Controls the CET settings for user mode and kernel mode respectively. MSR_IA32_PL{0,1,2,3}_SSP: Stores shadow stack pointers for CPL-0,1,2,3 protection respectively. MSR_IA32_INT_SSP_TAB: Stores base address of shadow stack pointer table. Two XSAVES state bits are introduced for CET: IA32_XSS:[bit 11]: Control saving/restoring user mode CET states IA32_XSS:[bit 12]: Control saving/restoring kernel mode CET states. Six VMCS fields are introduced for CET: {HOST,GUEST}_S_CET: Stores CET settings for kernel mode. {HOST,GUEST}_SSP: Stores shadow stack pointer of current task/thread. {HOST,GUEST}_INTR_SSP_TABLE: Stores base address of shadow stack pointer table. If VM_EXIT_LOAD_HOST_CET_STATE = 1, the host CET states are restored from the following VMCS fields at VM-Exit: HOST_S_CET HOST_SSP HOST_INTR_SSP_TABLE If VM_ENTRY_LOAD_GUEST_CET_STATE = 1, the guest CET states are loaded from the following VMCS fields at VM-Entry: GUEST_S_CET GUEST_SSP GUEST_INTR_SSP_TABLE Co-developed-by: Zhang Yi Z Signed-off-by: Zhang Yi Z Signed-off-by: Yang Weijiang Message-Id: <20210203113421.5759-6-weijiang.yang@intel.com> Signed-off-by: Paolo Bonzini --- arch/x86/include/asm/vmx.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h index c371ef695fcc..4e019fa968b8 100644 --- a/arch/x86/include/asm/vmx.h +++ b/arch/x86/include/asm/vmx.h @@ -102,6 +102,7 @@ #define VM_EXIT_CLEAR_BNDCFGS 0x00800000 #define VM_EXIT_PT_CONCEAL_PIP 0x01000000 #define VM_EXIT_CLEAR_IA32_RTIT_CTL 0x02000000 +#define VM_EXIT_LOAD_CET_STATE 0x10000000 #define VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR 0x00036dff @@ -115,6 +116,7 @@ #define VM_ENTRY_LOAD_BNDCFGS 0x00010000 #define VM_ENTRY_PT_CONCEAL_PIP 0x00020000 #define VM_ENTRY_LOAD_IA32_RTIT_CTL 0x00040000 +#define VM_ENTRY_LOAD_CET_STATE 0x00100000 #define VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR 0x000011ff @@ -343,6 +345,9 @@ enum vmcs_field { GUEST_PENDING_DBG_EXCEPTIONS = 0x00006822, GUEST_SYSENTER_ESP = 0x00006824, GUEST_SYSENTER_EIP = 0x00006826, + GUEST_S_CET = 0x00006828, + GUEST_SSP = 0x0000682a, + GUEST_INTR_SSP_TABLE = 0x0000682c, HOST_CR0 = 0x00006c00, HOST_CR3 = 0x00006c02, HOST_CR4 = 0x00006c04, @@ -355,6 +360,9 @@ enum vmcs_field { HOST_IA32_SYSENTER_EIP = 0x00006c12, HOST_RSP = 0x00006c14, HOST_RIP = 0x00006c16, + HOST_S_CET = 0x00006c18, + HOST_SSP = 0x00006c1a, + HOST_INTR_SSP_TABLE = 0x00006c1c }; /* From patchwork Thu Jun 16 08:46:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883578 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD125C43334 for ; Thu, 16 Jun 2022 08:49:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376404AbiFPItS (ORCPT ); Thu, 16 Jun 2022 04:49:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36604 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1359794AbiFPIs5 (ORCPT ); Thu, 16 Jun 2022 04:48:57 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 035A65EDED; Thu, 16 Jun 2022 01:47:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369271; x=1686905271; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ulzzAHI2dEL92o/KzFQRO8FgEw/C8l+obGu5qgBKRBU=; b=LaVn5YXpwgxFa42lbbjKSQ4XkGjxh21W01QUcMyUhR67jq4vC41/APw1 X6mI2WZ3rmr3tIAKuVna+Bavinku6DXKX6trvyZ38DYBgk3POLIPNqKXU ucOy7rt5lSKHy3N8qCPcxIoJWS2rVIB3kGMgCSen2ZTb86z4URVtOjglx RV21S0losdGclc/xhD2LH+q3DtjdBZ7yRI63rQsDcJ/x3T8gkl7hxtnWN kDZMX7BxVfQPK3d6dY8URjNEfsjPDffoFC6igjWIQtQrKxRMrNJPcYI5g feSsDbIwfYAk6Ntp6gBpdfpAzCK+6tAD9rk/6jexnfpjK+u2pgrohMTAb A==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664558" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664558" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083158" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Sean Christopherson Subject: [PATCH 11/19] KVM: x86: Add fault checks for CR4.CET Date: Thu, 16 Jun 2022 04:46:35 -0400 Message-Id: <20220616084643.19564-12-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add the fault checks for CR4.CET, which is the master control for all CET features (SHSTK and IBT). In addition to basic support checks, CET can be enabled if and only if CR0.WP==1, i.e. setting CR4.CET=1 faults if CR0.WP==0 and setting CR0.WP=0 fails if CR4.CET==1. Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Yang Weijiang Message-Id: <20210203113421.5759-7-weijiang.yang@intel.com> Signed-off-by: Paolo Bonzini --- arch/x86/kvm/x86.c | 6 ++++++ arch/x86/kvm/x86.h | 3 +++ 2 files changed, 9 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 40749e47cda7..cce789f1246a 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -952,6 +952,9 @@ int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) (is_64_bit_mode(vcpu) || kvm_read_cr4_bits(vcpu, X86_CR4_PCIDE))) return 1; + if (!(cr0 & X86_CR0_WP) && kvm_read_cr4_bits(vcpu, X86_CR4_CET)) + return 1; + static_call(kvm_x86_set_cr0)(vcpu, cr0); kvm_post_set_cr0(vcpu, old_cr0, cr0); @@ -1168,6 +1171,9 @@ int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) return 1; } + if ((cr4 & X86_CR4_CET) && !(kvm_read_cr0(vcpu) & X86_CR0_WP)) + return 1; + static_call(kvm_x86_set_cr4)(vcpu, cr4); kvm_post_set_cr4(vcpu, old_cr4, cr4); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index b9b1fff6d97a..01493b7ae150 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -477,6 +477,9 @@ bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); __reserved_bits |= X86_CR4_VMXE; \ if (!__cpu_has(__c, X86_FEATURE_PCID)) \ __reserved_bits |= X86_CR4_PCIDE; \ + if (!__cpu_has(__c, X86_FEATURE_SHSTK) && \ + !__cpu_has(__c, X86_FEATURE_IBT)) \ + __reserved_bits |= X86_CR4_CET; \ __reserved_bits; \ }) From patchwork Thu Jun 16 08:46:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE2B0C43334 for ; Thu, 16 Jun 2022 08:49:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376429AbiFPItA (ORCPT ); Thu, 16 Jun 2022 04:49:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376282AbiFPIsr (ORCPT ); Thu, 16 Jun 2022 04:48:47 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 20AF6140F4; Thu, 16 Jun 2022 01:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369262; x=1686905262; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JCzwZqsQLcwAChNZSNEP8E57myWOINezNqkUJ+d22/E=; b=E7T2NxjN1uDnhZWqhF/IfGuj6ZqE2BJdleQZLFtSGc+Wn7trzG4Keees 1qKyZpE0TYwql7WDDOKWxVYp+MSqPEejwWQxgxHk6WDNHh0RAajP1l15J gUbBVdxpy2fhz6d6fow7vxB+Itf6NGBTnXlr+ooqH3pcbABnsw/X9sXCJ 1pqh/PXifnd37YPqYKJ/C8VkY+TYZq8lDmqgUac8swMe/Uq37Pq0S7Xb4 P8HakVyZXt+kqyjFrYi9EX5ZRHPbQXn4XfFMYJmcuKWFsy7nF7DcnWZJ2 pSBP6B6hIhWJqcKeJ9SoDa+i/s37EhbRgxwkldSItuuLGeJcrC+klzRJ5 g==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259055230" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259055230" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083163" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Sean Christopherson Subject: [PATCH 12/19] KVM: VMX: Emulate reads and writes to CET MSRs Date: Thu, 16 Jun 2022 04:46:36 -0400 Message-Id: <20220616084643.19564-13-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add support for emulating read and write accesses to CET MSRs. CET MSRs are universally "special" as they are either context switched via dedicated VMCS fields or via XSAVES, i.e. no additional in-memory tracking is needed, but emulated reads/writes are more expensive. Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Yang Weijiang --- arch/x86/kvm/vmx/vmx.c | 42 ++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.h | 31 +++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 5e14e4c40007..d1f2ffa07576 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -1767,6 +1767,26 @@ static int vmx_get_msr_feature(struct kvm_msr_entry *msr) } } +static bool cet_is_msr_accessible(struct kvm_vcpu *vcpu, + struct msr_data *msr) +{ + if (!kvm_cet_user_supported()) + return false; + + if (msr->host_initiated) + return true; + + if (!guest_cpuid_has(vcpu, X86_FEATURE_SHSTK) && + !guest_cpuid_has(vcpu, X86_FEATURE_IBT)) + return false; + + if (msr->index == MSR_IA32_PL3_SSP && + !guest_cpuid_has(vcpu, X86_FEATURE_SHSTK)) + return false; + + return true; +} + /* * Reads an msr value (of 'msr_info->index') into 'msr_info->data'. * Returns 0 on success, non-0 otherwise. @@ -1906,6 +1926,12 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) else msr_info->data = vmx->pt_desc.guest.addr_a[index / 2]; break; + case MSR_IA32_U_CET: + case MSR_IA32_PL3_SSP: + if (!cet_is_msr_accessible(vcpu, msr_info)) + return 1; + kvm_get_xsave_msr(msr_info); + break; case MSR_IA32_DEBUGCTLMSR: msr_info->data = vmcs_read64(GUEST_IA32_DEBUGCTL); break; @@ -2238,6 +2264,22 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) else vmx->pt_desc.guest.addr_a[index / 2] = data; break; + case MSR_IA32_U_CET: + if (!cet_is_msr_accessible(vcpu, msr_info)) + return 1; + if ((data & GENMASK(9, 6)) || + is_noncanonical_address(data, vcpu)) + return 1; + kvm_set_xsave_msr(msr_info); + break; + case MSR_IA32_PL3_SSP: + if (!cet_is_msr_accessible(vcpu, msr_info)) + return 1; + if ((data & GENMASK(2, 0)) || + is_noncanonical_address(data, vcpu)) + return 1; + kvm_set_xsave_msr(msr_info); + break; case MSR_IA32_PERF_CAPABILITIES: if (data && !vcpu_to_pmu(vcpu)->version) return 1; diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 01493b7ae150..f6000e3fb195 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -2,6 +2,7 @@ #ifndef ARCH_X86_KVM_X86_H #define ARCH_X86_KVM_X86_H +#include #include #include #include @@ -323,6 +324,16 @@ static inline bool kvm_mpx_supported(void) == (XFEATURE_MASK_BNDREGS | XFEATURE_MASK_BNDCSR); } +/* + * Guest CET user mode states depend on host XSAVES/XRSTORS to save/restore + * when vCPU enter/exit user space. If host doesn't support CET user bit in + * XSS msr, then treat this case as KVM doesn't support CET user mode. + */ +static inline bool kvm_cet_user_supported(void) +{ + return !!(kvm_caps.supported_xss & XFEATURE_MASK_CET_USER); +} + extern unsigned int min_timer_period_us; extern bool enable_vmware_backdoor; @@ -491,4 +502,24 @@ int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size, unsigned int port, void *data, unsigned int count, int in); +/* + * We've already loaded guest MSRs in __msr_io() when check the MSR index. + * In case vcpu has been preempted, we need to disable preemption, check + * and reload the guest fpu states before issue MSR read/write, + * fpu_lock_and_load() serves the purpose well. + */ +static inline void kvm_get_xsave_msr(struct msr_data *msr_info) +{ + fpu_lock_and_load(); + rdmsrl(msr_info->index, msr_info->data); + fpregs_unlock(); +} + +static inline void kvm_set_xsave_msr(struct msr_data *msr_info) +{ + fpu_lock_and_load(); + wrmsrl(msr_info->index, msr_info->data); + fpregs_unlock(); +} + #endif From patchwork Thu Jun 16 08:46:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD3FCC43334 for ; Thu, 16 Jun 2022 08:49:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1359865AbiFPIt0 (ORCPT ); Thu, 16 Jun 2022 04:49:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376354AbiFPIsw (ORCPT ); Thu, 16 Jun 2022 04:48:52 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 626551A80F; Thu, 16 Jun 2022 01:47:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369264; x=1686905264; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=wMy9QLp1MqvMyYPb5VEt2CgYty+Gh8hHunwvtTY1ADI=; b=SFiyhX1TSpeiikyvDfN4djnFzygwVy/Z3vEO8v1EEVRgpvkGsXfzdU2j 7Dikzay2mjhLnEhDAMilrTnMoyTB324/VN/in4wBPVyPapJuw/QZeTAiq Q/lGJVF9LzS6OO3AFZfcAOUM7EJhY0s22OwdwX/bpd/fCO2RCQ1HghnG8 udefnEoH0V5eq1Mr31QlmistxOwUVncirD37Xi730Qm+Lr6p4lKYOCApV mehrI1IMCTqjDoMOtgU0vUF3zwnXXK9N12GdNcT5PP9n+fRxXtP5FeI2h rbvwiQCvJ2CShKFfTUZi2XLp8WsXggmASaM68lnDyGEhT/5BZ1uvY7hz7 A==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259055231" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259055231" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083165" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Sean Christopherson Subject: [PATCH 13/19] KVM: VMX: Add a synthetic MSR to allow userspace VMM to access GUEST_SSP Date: Thu, 16 Jun 2022 04:46:37 -0400 Message-Id: <20220616084643.19564-14-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Introduce a host-only synthetic MSR, MSR_KVM_GUEST_SSP so that the VMM can read/write the guest's SSP, e.g. to migrate CET state. Use a synthetic MSR, e.g. as opposed to a VCPU_REG_, as GUEST_SSP is subject to the same consistency checks as the PL*_SSP MSRs, i.e. can share code. Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Yang Weijiang --- arch/x86/include/uapi/asm/kvm_para.h | 1 + arch/x86/kvm/vmx/vmx.c | 15 ++++++++++++--- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/uapi/asm/kvm_para.h b/arch/x86/include/uapi/asm/kvm_para.h index 6e64b27b2c1e..7af465e4e0bd 100644 --- a/arch/x86/include/uapi/asm/kvm_para.h +++ b/arch/x86/include/uapi/asm/kvm_para.h @@ -58,6 +58,7 @@ #define MSR_KVM_ASYNC_PF_INT 0x4b564d06 #define MSR_KVM_ASYNC_PF_ACK 0x4b564d07 #define MSR_KVM_MIGRATION_CONTROL 0x4b564d08 +#define MSR_KVM_GUEST_SSP 0x4b564d09 struct kvm_steal_time { __u64 steal; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index d1f2ffa07576..fc1229f23987 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -1780,7 +1780,8 @@ static bool cet_is_msr_accessible(struct kvm_vcpu *vcpu, !guest_cpuid_has(vcpu, X86_FEATURE_IBT)) return false; - if (msr->index == MSR_IA32_PL3_SSP && + if ((msr->index == MSR_IA32_PL3_SSP || + msr->index == MSR_KVM_GUEST_SSP) && !guest_cpuid_has(vcpu, X86_FEATURE_SHSTK)) return false; @@ -1928,9 +1929,13 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) break; case MSR_IA32_U_CET: case MSR_IA32_PL3_SSP: + case MSR_KVM_GUEST_SSP: if (!cet_is_msr_accessible(vcpu, msr_info)) return 1; - kvm_get_xsave_msr(msr_info); + if (msr_info->index == MSR_KVM_GUEST_SSP) + msr_info->data = vmcs_readl(GUEST_SSP); + else + kvm_get_xsave_msr(msr_info); break; case MSR_IA32_DEBUGCTLMSR: msr_info->data = vmcs_read64(GUEST_IA32_DEBUGCTL); @@ -2273,12 +2278,16 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) kvm_set_xsave_msr(msr_info); break; case MSR_IA32_PL3_SSP: + case MSR_KVM_GUEST_SSP: if (!cet_is_msr_accessible(vcpu, msr_info)) return 1; if ((data & GENMASK(2, 0)) || is_noncanonical_address(data, vcpu)) return 1; - kvm_set_xsave_msr(msr_info); + if (msr_index == MSR_KVM_GUEST_SSP) + vmcs_writel(GUEST_SSP, data); + else + kvm_set_xsave_msr(msr_info); break; case MSR_IA32_PERF_CAPABILITIES: if (data && !vcpu_to_pmu(vcpu)->version) From patchwork Thu Jun 16 08:46:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E91E8C433EF for ; Thu, 16 Jun 2022 08:49:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376427AbiFPIto (ORCPT ); Thu, 16 Jun 2022 04:49:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1359855AbiFPIs6 (ORCPT ); Thu, 16 Jun 2022 04:48:58 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4D925F247; Thu, 16 Jun 2022 01:47:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369276; x=1686905276; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=5ds9yOVSoDWedK4bluyWKuKvK1FziQP9xxqZoYWM0jw=; b=PCt42bnYdfFn5fkac4tb0NEnghYnSOvvq4IKlfCtz9jbxSvCpAD5D6sP SdOOgouY4cSNW+QCzcxeUBY2Sj/9Yg6lm+eWs6rRGgAGLJRrV/GOPs1Ky 79JAEMYL8ZKAMhDjUaCSADR719WNgxWApxK2RMN2tmpOcqs0q+3KrZOgK rF/JaBeWtbBtB99ihNzbfbY5u3SjwBBhuB0qjsWslSVqxt4Ycc6qmj148 rgrFwbP1JAC0Ubf+GNbtG7Rz+K4WXj6/55YWMeJweGIQuGB3vV4XySZV+ TDW1V4zS/IeT5hWNFwQYJj4nlpZyfa2Zf9fAtVlY29IkLxt4ye9COVRqX g==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664561" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664561" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083169" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Sean Christopherson Subject: [PATCH 14/19] KVM: x86: Report CET MSRs as to-be-saved if CET is supported Date: Thu, 16 Jun 2022 04:46:38 -0400 Message-Id: <20220616084643.19564-15-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Report all CET MSRs, including the synthetic GUEST_SSP MSR, as to-be-saved, e.g. for migration, if CET is supported by KVM. Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Yang Weijiang --- arch/x86/kvm/x86.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index cce789f1246a..3613b73f13fb 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1460,6 +1460,7 @@ static const u32 msrs_to_save_all[] = { MSR_F15H_PERF_CTR3, MSR_F15H_PERF_CTR4, MSR_F15H_PERF_CTR5, MSR_IA32_XFD, MSR_IA32_XFD_ERR, MSR_IA32_XSS, + MSR_IA32_U_CET, MSR_IA32_PL3_SSP, MSR_KVM_GUEST_SSP, }; static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_all)]; @@ -6814,6 +6815,12 @@ static void kvm_init_msr_list(void) if (!kvm_caps.supported_xss) continue; break; + case MSR_KVM_GUEST_SSP: + case MSR_IA32_U_CET: + case MSR_IA32_PL3_SSP: + if (!kvm_cet_user_supported()) + continue; + break; default: break; } From patchwork Thu Jun 16 08:46:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F20B1C43334 for ; Thu, 16 Jun 2022 08:49:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376284AbiFPIt2 (ORCPT ); Thu, 16 Jun 2022 04:49:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36368 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376350AbiFPIsw (ORCPT ); Thu, 16 Jun 2022 04:48:52 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3BA3C2DAAC; Thu, 16 Jun 2022 01:47:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369265; x=1686905265; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=R7sGqyE6ELUI6BrR6Bhsf2tb+qtJTh756ZHNEPItrck=; b=fLK0PIDFI+JmM2r9K/0fKqOD0zEp6kXOiiHl6n9tKaCzWFoyrxR+ikfE +WKahze3PSD6XqJo1eLtsy5aLfwyMzkCelveRKeYh9RAGCI8d08s+7CX7 JiqV6iQARyG0LWij+qGGFNIRLkkRXUkRiwWonwdtATuyNyMafO6yWsE/O BYvoQjSMvKIjP4uKqee2Kw5Vaj42QHF9HnL1eDE2h/+tSQ0cwxyQUBc4b GUcofB1bLKhUN2KJCvMA+MxKh/onTicxcFbbu4Y81uHr+Wj0KW0W5Wyfq H8GNynAMZ5IHx68be9vWzt4iZmvhBIkz4AwK06Owp+zXlqNwEUxLmU5zF w==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259055233" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259055233" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:41 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083172" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com Subject: [PATCH 15/19] KVM: x86: Save/Restore GUEST_SSP to/from SMM state save area Date: Thu, 16 Jun 2022 04:46:39 -0400 Message-Id: <20220616084643.19564-16-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Save GUEST_SSP in the SMM state save area when guest exits to SMM due to SMI and restore it when guest exits SMM. Signed-off-by: Yang Weijiang Message-Id: <20210203113421.5759-15-weijiang.yang@intel.com> [Change the SMM offset to some place that is actually free. - Paolo] Signed-off-by: Paolo Bonzini --- arch/x86/kvm/emulate.c | 11 +++++++++++ arch/x86/kvm/x86.c | 10 ++++++++++ 2 files changed, 21 insertions(+) diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index 39ea9138224c..eb0d45ae5214 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -2558,6 +2558,17 @@ static int rsm_load_state_64(struct x86_emulate_ctxt *ctxt, return r; } + if (kvm_cet_user_supported()) { + struct msr_data msr; + + val = GET_SMSTATE(u64, smstate, 0x7f08); + msr.index = MSR_KVM_GUEST_SSP; + msr.host_initiated = true; + msr.data = val; + /* Mimic host_initiated access to bypass ssp access check. */ + kvm_x86_ops.set_msr(ctxt->vcpu, &msr); + } + return X86EMUL_CONTINUE; } #endif diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 3613b73f13fb..86bccb12f036 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9833,6 +9833,16 @@ static void enter_smm_save_state_64(struct kvm_vcpu *vcpu, char *buf) for (i = 0; i < 6; i++) enter_smm_save_seg_64(vcpu, buf, i); + + if (kvm_cet_user_supported()) { + struct msr_data msr; + + msr.index = MSR_KVM_GUEST_SSP; + msr.host_initiated = true; + /* GUEST_SSP is stored in VMCS at vm-exit. */ + kvm_x86_ops.get_msr(vcpu, &msr); + put_smstate(u64, buf, 0x7f08, msr.data); + } } #endif From patchwork Thu Jun 16 08:46:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883590 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E345DCCA47D for ; Thu, 16 Jun 2022 08:49:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376557AbiFPItr (ORCPT ); Thu, 16 Jun 2022 04:49:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35622 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376414AbiFPIs7 (ORCPT ); Thu, 16 Jun 2022 04:48:59 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 178575F250; Thu, 16 Jun 2022 01:47:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369278; x=1686905278; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=72vrd16dybfGhayhrx0NsJFobw9Ileac81SFXF0QanY=; b=mfJxO5qV2p6QrcqBExhIGUxBOFPVtYB/KhiZf0nuA3V8Efeljt+n1NF5 lEXpq9f4xDQN8f6tLaxUtJEk1ciGBbeEuYiaegdHzbF1Q0k/TjZ8hSakb bqF4KrI71HnrGWso38+FYKQmv5GEK4pW1M8dS0OkV0ItSMLvMi6r9CpJJ HEoj/RfPlSw35S2vzEh9yXkvab9ufmAQxncatV1N48t30Mi+ypugIEEqS 7cVrOCDhtb89DfMNIPtBBhGQLNYQAo+JwURzoEDTr23c5/AFqUNdhbY81 bsvjmRRI1MURxLub5FWZ7epuCg8R4irqqwsgRM6V4o+vAKCasRUPUmUue w==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664562" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664562" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:42 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083175" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Sean Christopherson Subject: [PATCH 16/19] KVM: x86: Enable CET virtualization for VMX and advertise CET to userspace Date: Thu, 16 Jun 2022 04:46:40 -0400 Message-Id: <20220616084643.19564-17-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Set the feature bits so that CET capabilities can be seen in guest via CPUID enumeration. Add CR4.CET bit support in order to allow guest set CET master control bit(CR4.CET). Disable KVM CET feature if unrestricted_guest is unsupported/disabled as KVM does not support emulating CET. Don't expose CET feature if dependent CET bits are cleared in host XSS, or if XSAVES isn't supported. Updating the CET features in common x86 is a little ugly, but there is no clean solution without risking breakage of SVM if SVM hardware ever gains support for CET, e.g. moving everything to common x86 would prematurely expose CET on SVM. The alternative is to put all the logic in VMX, but that means rereading host_xss in VMX and duplicating the XSAVES check across VMX and SVM. Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Yang Weijiang --- arch/x86/include/asm/kvm_host.h | 3 ++- arch/x86/kvm/cpuid.c | 5 +++-- arch/x86/kvm/vmx/capabilities.h | 4 ++++ arch/x86/kvm/vmx/vmx.c | 37 ++++++++++++++++++++++++++++----- arch/x86/kvm/x86.c | 21 ++++++++++++++++++- 5 files changed, 61 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 7e98b2876380..a7e5463d0107 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -120,7 +120,8 @@ | X86_CR4_PGE | X86_CR4_PCE | X86_CR4_OSFXSR | X86_CR4_PCIDE \ | X86_CR4_OSXSAVE | X86_CR4_SMEP | X86_CR4_FSGSBASE \ | X86_CR4_OSXMMEXCPT | X86_CR4_LA57 | X86_CR4_VMXE \ - | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP)) + | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP \ + | X86_CR4_CET)) #define CR8_RESERVED_BITS (~(unsigned long)X86_CR8_TPR) diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 46ca0f1abbcb..12d527e612e5 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -581,7 +581,7 @@ void kvm_set_cpu_caps(void) F(AVX512_VPOPCNTDQ) | F(UMIP) | F(AVX512_VBMI2) | F(GFNI) | F(VAES) | F(VPCLMULQDQ) | F(AVX512_VNNI) | F(AVX512_BITALG) | F(CLDEMOTE) | F(MOVDIRI) | F(MOVDIR64B) | 0 /*WAITPKG*/ | - F(SGX_LC) | F(BUS_LOCK_DETECT) + F(SGX_LC) | F(BUS_LOCK_DETECT) | F(SHSTK) ); /* Set LA57 based on hardware capability. */ if (cpuid_ecx(7) & F(LA57)) @@ -599,7 +599,8 @@ void kvm_set_cpu_caps(void) F(SPEC_CTRL_SSBD) | F(ARCH_CAPABILITIES) | F(INTEL_STIBP) | F(MD_CLEAR) | F(AVX512_VP2INTERSECT) | F(FSRM) | F(SERIALIZE) | F(TSXLDTRK) | F(AVX512_FP16) | - F(AMX_TILE) | F(AMX_INT8) | F(AMX_BF16) + F(AMX_TILE) | F(AMX_INT8) | F(AMX_BF16) | + F(IBT) ); /* TSC_ADJUST and ARCH_CAPABILITIES are emulated in software. */ diff --git a/arch/x86/kvm/vmx/capabilities.h b/arch/x86/kvm/vmx/capabilities.h index 069d8d298e1d..6849888f7b46 100644 --- a/arch/x86/kvm/vmx/capabilities.h +++ b/arch/x86/kvm/vmx/capabilities.h @@ -106,6 +106,10 @@ static inline bool cpu_has_load_perf_global_ctrl(void) return vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL; } +static inline bool cpu_has_load_cet_ctrl(void) +{ + return (vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_CET_STATE); +} static inline bool cpu_has_vmx_mpx(void) { return vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_BNDCFGS; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index fc1229f23987..4bdede87669a 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -2516,6 +2516,7 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf, { VM_ENTRY_LOAD_IA32_EFER, VM_EXIT_LOAD_IA32_EFER }, { VM_ENTRY_LOAD_BNDCFGS, VM_EXIT_CLEAR_BNDCFGS }, { VM_ENTRY_LOAD_IA32_RTIT_CTL, VM_EXIT_CLEAR_IA32_RTIT_CTL }, + { VM_ENTRY_LOAD_CET_STATE, VM_EXIT_LOAD_CET_STATE }, }; memset(vmcs_conf, 0, sizeof(*vmcs_conf)); @@ -2636,7 +2637,8 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf, VM_EXIT_LOAD_IA32_EFER | VM_EXIT_CLEAR_BNDCFGS | VM_EXIT_PT_CONCEAL_PIP | - VM_EXIT_CLEAR_IA32_RTIT_CTL; + VM_EXIT_CLEAR_IA32_RTIT_CTL | + VM_EXIT_LOAD_CET_STATE; if (adjust_vmx_controls(min, opt, MSR_IA32_VMX_EXIT_CTLS, &_vmexit_control) < 0) return -EIO; @@ -2660,7 +2662,8 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf, VM_ENTRY_LOAD_IA32_EFER | VM_ENTRY_LOAD_BNDCFGS | VM_ENTRY_PT_CONCEAL_PIP | - VM_ENTRY_LOAD_IA32_RTIT_CTL; + VM_ENTRY_LOAD_IA32_RTIT_CTL | + VM_ENTRY_LOAD_CET_STATE; if (adjust_vmx_controls(min, opt, MSR_IA32_VMX_ENTRY_CTLS, &_vmentry_control) < 0) return -EIO; @@ -2705,7 +2708,6 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf, } } - rdmsr(MSR_IA32_VMX_BASIC, vmx_msr_low, vmx_msr_high); /* IA-32 SDM Vol 3B: VMCS size is never greater than 4kB. */ @@ -6159,6 +6161,12 @@ void dump_vmcs(struct kvm_vcpu *vcpu) if (vmcs_read32(VM_EXIT_MSR_STORE_COUNT) > 0) vmx_dump_msrs("guest autostore", &vmx->msr_autostore.guest); + if (vmentry_ctl & VM_ENTRY_LOAD_CET_STATE) { + pr_err("S_CET = 0x%016lx\n", vmcs_readl(GUEST_S_CET)); + pr_err("SSP = 0x%016lx\n", vmcs_readl(GUEST_SSP)); + pr_err("SSP TABLE = 0x%016lx\n", + vmcs_readl(GUEST_INTR_SSP_TABLE)); + } pr_err("*** Host State ***\n"); pr_err("RIP = 0x%016lx RSP = 0x%016lx\n", vmcs_readl(HOST_RIP), vmcs_readl(HOST_RSP)); @@ -6236,6 +6244,12 @@ void dump_vmcs(struct kvm_vcpu *vcpu) if (secondary_exec_control & SECONDARY_EXEC_ENABLE_VPID) pr_err("Virtual processor ID = 0x%04x\n", vmcs_read16(VIRTUAL_PROCESSOR_ID)); + if (vmexit_ctl & VM_EXIT_LOAD_CET_STATE) { + pr_err("S_CET = 0x%016lx\n", vmcs_readl(HOST_S_CET)); + pr_err("SSP = 0x%016lx\n", vmcs_readl(HOST_SSP)); + pr_err("SSP TABLE = 0x%016lx\n", + vmcs_readl(HOST_INTR_SSP_TABLE)); + } } /* @@ -7679,9 +7693,10 @@ static __init void vmx_set_cpu_caps(void) kvm_cpu_cap_set(X86_FEATURE_UMIP); /* CPUID 0xD.1 */ - kvm_caps.supported_xss = 0; - if (!cpu_has_vmx_xsaves()) + if (!cpu_has_vmx_xsaves()) { kvm_cpu_cap_clear(X86_FEATURE_XSAVES); + kvm_caps.supported_xss = 0; + } /* CPUID 0x80000001 and 0x7 (RDPID) */ if (!cpu_has_vmx_rdtscp()) { @@ -7691,6 +7706,18 @@ static __init void vmx_set_cpu_caps(void) if (cpu_has_vmx_waitpkg()) kvm_cpu_cap_check_and_set(X86_FEATURE_WAITPKG); + + if (!cpu_has_load_cet_ctrl() || !enable_unrestricted_guest) { + kvm_cpu_cap_clear(X86_FEATURE_SHSTK); + kvm_cpu_cap_clear(X86_FEATURE_IBT); + kvm_caps.supported_xss &= ~XFEATURE_MASK_CET_USER; + } + +#ifndef CONFIG_X86_SHADOW_STACK + if (boot_cpu_has(X86_FEATURE_SHSTK)) + kvm_cpu_cap_clear(X86_FEATURE_SHSTK); +#endif + } static void vmx_request_immediate_exit(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 86bccb12f036..fe049d0e5ecc 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -217,6 +217,8 @@ static struct kvm_user_return_msrs __percpu *user_return_msrs; | XFEATURE_MASK_BNDCSR | XFEATURE_MASK_AVX512 \ | XFEATURE_MASK_PKRU | XFEATURE_MASK_XTILE) +#define KVM_SUPPORTED_XSS (XFEATURE_MASK_CET_USER) + u64 __read_mostly host_efer; EXPORT_SYMBOL_GPL(host_efer); @@ -11807,8 +11809,10 @@ int kvm_arch_hardware_setup(void *opaque) rdmsrl_safe(MSR_EFER, &host_efer); - if (boot_cpu_has(X86_FEATURE_XSAVES)) + if (boot_cpu_has(X86_FEATURE_XSAVES)) { rdmsrl(MSR_IA32_XSS, host_xss); + kvm_caps.supported_xss = host_xss & KVM_SUPPORTED_XSS; + } kvm_init_pmu_capability(); @@ -11823,6 +11827,21 @@ int kvm_arch_hardware_setup(void *opaque) if (!kvm_cpu_cap_has(X86_FEATURE_XSAVES)) kvm_caps.supported_xss = 0; + /* Update CET features now as kvm_caps.supported_xss is finalized. */ + if (!kvm_cet_user_supported()) { + kvm_cpu_cap_clear(X86_FEATURE_SHSTK); + kvm_cpu_cap_clear(X86_FEATURE_IBT); + } + + /* + * If SHSTK and IBT are disabled either by user space or unselection in + * Kconfig,then the feature bits should have been removed from KVM caps + * by this point, clear CET user bit in kvm_caps.supported_xss too. + */ + if (!kvm_cpu_cap_has(X86_FEATURE_SHSTK) && + !kvm_cpu_cap_has(X86_FEATURE_IBT)) + kvm_caps.supported_xss &= ~XFEATURE_CET_USER; + #define __kvm_cpu_cap_has(UNUSED_, f) kvm_cpu_cap_has(f) cr4_reserved_bits = __cr4_reserved_bits(__kvm_cpu_cap_has, UNUSED_); #undef __kvm_cpu_cap_has From patchwork Thu Jun 16 08:46:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883584 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4576CCA47F for ; Thu, 16 Jun 2022 08:49:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376372AbiFPIte (ORCPT ); Thu, 16 Jun 2022 04:49:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36532 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376392AbiFPIsy (ORCPT ); Thu, 16 Jun 2022 04:48:54 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6035A5DE4A; Thu, 16 Jun 2022 01:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369268; x=1686905268; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Ztaj5gfZsCYVHfksYf+7zuhCQauSNN6mu9/n9Y3T3iQ=; b=Vj7pxj1McicP+r4f+895P/n+IAYgV2c4Ah3Q7aTrFQYXMaXdpvXUb+D2 OYyAjYyiP3/nfNQClYqOGDet9DUZ7prPe4pkoJbauMYOIwBUhTkBnR1R6 eie8JnYyKR0RsQZwDIC38mS4iYsdm3IosDoojMKwK2HZ5N46AgBCR3WqS 1lkI+J5xw69U8KnToBdJyPQiKQIkwcZCXfpsKQUJ3Cj3Rp6asFI4P3ZGB CaeTMNLP/za+5tNIFlvp1HKZDJqxyiVLKoTJZaBARXr7csFe6hbvKWW1u rY9F5DrcclI/UfIz79PCD76Jesyol5UqseP1WNDNRpjNwdNmjEusP5UJq A==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259055234" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259055234" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:42 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083178" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Zhang Yi Z , Sean Christopherson Subject: [PATCH 17/19] KVM: VMX: Pass through CET MSRs to the guest when supported Date: Thu, 16 Jun 2022 04:46:41 -0400 Message-Id: <20220616084643.19564-18-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Pass through CET user mode MSRs when the associated CET component is enabled to improve guest performance. All CET MSRs are context switched, either via dedicated VMCS fields or XSAVES. Co-developed-by: Zhang Yi Z Signed-off-by: Zhang Yi Z Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Yang Weijiang --- arch/x86/kvm/vmx/vmx.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 4bdede87669a..9aebd67ff03e 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -583,6 +583,9 @@ static bool is_valid_passthrough_msr(u32 msr) case MSR_LBR_CORE_TO ... MSR_LBR_CORE_TO + 8: /* LBR MSRs. These are handled in vmx_update_intercept_for_lbr_msrs() */ return true; + case MSR_IA32_U_CET: + case MSR_IA32_PL3_SSP: + return true; } r = possible_passthrough_msr_slot(msr) != -ENOENT; @@ -7595,6 +7598,23 @@ static void update_intel_pt_cfg(struct kvm_vcpu *vcpu) vmx->pt_desc.ctl_bitmask &= ~(0xfULL << (32 + i * 4)); } +static bool is_cet_state_supported(struct kvm_vcpu *vcpu, u32 xss_state) +{ + return (kvm_caps.supported_xss & xss_state) && + (guest_cpuid_has(vcpu, X86_FEATURE_SHSTK) || + guest_cpuid_has(vcpu, X86_FEATURE_IBT)); +} + +static void vmx_update_intercept_for_cet_msr(struct kvm_vcpu *vcpu) +{ + bool incpt = !is_cet_state_supported(vcpu, XFEATURE_MASK_CET_USER); + + vmx_set_intercept_for_msr(vcpu, MSR_IA32_U_CET, MSR_TYPE_RW, incpt); + + incpt |= !guest_cpuid_has(vcpu, X86_FEATURE_SHSTK); + vmx_set_intercept_for_msr(vcpu, MSR_IA32_PL3_SSP, MSR_TYPE_RW, incpt); +} + static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); @@ -7657,6 +7677,9 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) /* Refresh #PF interception to account for MAXPHYADDR changes. */ vmx_update_exception_bitmap(vcpu); + + if (kvm_cet_user_supported()) + vmx_update_intercept_for_cet_msr(vcpu); } static __init void vmx_set_cpu_caps(void) From patchwork Thu Jun 16 08:46:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883588 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9962AC43334 for ; Thu, 16 Jun 2022 08:49:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376290AbiFPItm (ORCPT ); Thu, 16 Jun 2022 04:49:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1359860AbiFPIs6 (ORCPT ); Thu, 16 Jun 2022 04:48:58 -0400 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 177DB5F24E; Thu, 16 Jun 2022 01:47:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369278; x=1686905278; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=p5k/MrqOyMSrhK6Fesac4BN03NYVV73OQspmMYtoAHw=; b=fAz4mctTkRZIdSjnDKYjTSLZS8a3jzc06P7uOoXB9HJbrJUGgfahOus0 n7MN47jtKowtR+JkoGY/+tt9c7gRx6GgoOhPmAlI4SPL07DJQzs7oED+x M69/1l976RwdV7tX6lAqvx5Ac2CYOcAznFfkMeLAqqg1ay4lOIckjvoHi 5kCPLsKSEobjqsazWEU3y52TYT8U6l5rXXXtOmJ0vLh7htXYwJ1fndJtU 6xTJk3qQbpgXadLLh0/nb+uG3sspSMmiho1g3SD8MRGngZPC1N0Y0rkaW Md49NBe/IGjGMPsyT3r0dAw+GybSLGAjaWMIUEmgiZuYI86s0MD3JgoFT A==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259664563" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259664563" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:42 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083181" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com, Sean Christopherson Subject: [PATCH 18/19] KVM: nVMX: Enable CET support for nested VMX Date: Thu, 16 Jun 2022 04:46:42 -0400 Message-Id: <20220616084643.19564-19-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add vmcs12 fields for all CET fields, pass-through CET MSRs to L2 when possible, and enumerate the VMCS controls and CR4 bit as supported. Co-developed-by: Sean Christopherson Signed-off-by: Sean Christopherson Signed-off-by: Yang Weijiang --- arch/x86/kvm/vmx/nested.c | 14 ++++++++++++-- arch/x86/kvm/vmx/vmcs12.c | 6 ++++++ arch/x86/kvm/vmx/vmcs12.h | 14 +++++++++++++- arch/x86/kvm/vmx/vmx.c | 2 ++ 4 files changed, 33 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 01fe23c6fa49..f31f3d394507 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -684,6 +684,13 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu, nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0, MSR_IA32_PRED_CMD, MSR_TYPE_W); + /* Pass CET MSRs to nested VM if L0 and L1 are set to pass-through. */ + nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0, + MSR_IA32_U_CET, MSR_TYPE_RW); + + nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0, + MSR_IA32_PL3_SSP, MSR_TYPE_RW); + kvm_vcpu_unmap(vcpu, &vmx->nested.msr_bitmap_map, false); vmx->nested.force_msr_bitmap_recalc = false; @@ -6593,7 +6600,9 @@ void nested_vmx_setup_ctls_msrs(struct nested_vmx_msrs *msrs, u32 ept_caps) VM_EXIT_HOST_ADDR_SPACE_SIZE | #endif VM_EXIT_LOAD_IA32_PAT | VM_EXIT_SAVE_IA32_PAT | - VM_EXIT_CLEAR_BNDCFGS | VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL; + VM_EXIT_CLEAR_BNDCFGS | VM_EXIT_LOAD_IA32_PERF_GLOBAL_CTRL | + VM_EXIT_LOAD_CET_STATE; + msrs->exit_ctls_high |= VM_EXIT_ALWAYSON_WITHOUT_TRUE_MSR | VM_EXIT_LOAD_IA32_EFER | VM_EXIT_SAVE_IA32_EFER | @@ -6613,7 +6622,8 @@ void nested_vmx_setup_ctls_msrs(struct nested_vmx_msrs *msrs, u32 ept_caps) VM_ENTRY_IA32E_MODE | #endif VM_ENTRY_LOAD_IA32_PAT | VM_ENTRY_LOAD_BNDCFGS | - VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL; + VM_ENTRY_LOAD_IA32_PERF_GLOBAL_CTRL | VM_ENTRY_LOAD_CET_STATE; + msrs->entry_ctls_high |= (VM_ENTRY_ALWAYSON_WITHOUT_TRUE_MSR | VM_ENTRY_LOAD_IA32_EFER); diff --git a/arch/x86/kvm/vmx/vmcs12.c b/arch/x86/kvm/vmx/vmcs12.c index 2251b60920f8..4c3836db548e 100644 --- a/arch/x86/kvm/vmx/vmcs12.c +++ b/arch/x86/kvm/vmx/vmcs12.c @@ -138,6 +138,9 @@ const unsigned short vmcs12_field_offsets[] = { FIELD(GUEST_PENDING_DBG_EXCEPTIONS, guest_pending_dbg_exceptions), FIELD(GUEST_SYSENTER_ESP, guest_sysenter_esp), FIELD(GUEST_SYSENTER_EIP, guest_sysenter_eip), + FIELD(GUEST_S_CET, guest_s_cet), + FIELD(GUEST_SSP, guest_ssp), + FIELD(GUEST_INTR_SSP_TABLE, guest_ssp_tbl), FIELD(HOST_CR0, host_cr0), FIELD(HOST_CR3, host_cr3), FIELD(HOST_CR4, host_cr4), @@ -150,5 +153,8 @@ const unsigned short vmcs12_field_offsets[] = { FIELD(HOST_IA32_SYSENTER_EIP, host_ia32_sysenter_eip), FIELD(HOST_RSP, host_rsp), FIELD(HOST_RIP, host_rip), + FIELD(HOST_S_CET, host_s_cet), + FIELD(HOST_SSP, host_ssp), + FIELD(HOST_INTR_SSP_TABLE, host_ssp_tbl), }; const unsigned int nr_vmcs12_fields = ARRAY_SIZE(vmcs12_field_offsets); diff --git a/arch/x86/kvm/vmx/vmcs12.h b/arch/x86/kvm/vmx/vmcs12.h index 746129ddd5ae..672abd1f500b 100644 --- a/arch/x86/kvm/vmx/vmcs12.h +++ b/arch/x86/kvm/vmx/vmcs12.h @@ -117,7 +117,13 @@ struct __packed vmcs12 { natural_width host_ia32_sysenter_eip; natural_width host_rsp; natural_width host_rip; - natural_width paddingl[8]; /* room for future expansion */ + natural_width host_s_cet; + natural_width host_ssp; + natural_width host_ssp_tbl; + natural_width guest_s_cet; + natural_width guest_ssp; + natural_width guest_ssp_tbl; + natural_width paddingl[2]; /* room for future expansion */ u32 pin_based_vm_exec_control; u32 cpu_based_vm_exec_control; u32 exception_bitmap; @@ -293,6 +299,12 @@ static inline void vmx_check_vmcs12_offsets(void) CHECK_OFFSET(host_ia32_sysenter_eip, 656); CHECK_OFFSET(host_rsp, 664); CHECK_OFFSET(host_rip, 672); + CHECK_OFFSET(host_s_cet, 680); + CHECK_OFFSET(host_ssp, 688); + CHECK_OFFSET(host_ssp_tbl, 696); + CHECK_OFFSET(guest_s_cet, 704); + CHECK_OFFSET(guest_ssp, 712); + CHECK_OFFSET(guest_ssp_tbl, 720); CHECK_OFFSET(pin_based_vm_exec_control, 744); CHECK_OFFSET(cpu_based_vm_exec_control, 748); CHECK_OFFSET(exception_bitmap, 752); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 9aebd67ff03e..00782d1750a5 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7508,6 +7508,8 @@ static void nested_vmx_cr_fixed1_bits_update(struct kvm_vcpu *vcpu) cr4_fixed1_update(X86_CR4_PKE, ecx, feature_bit(PKU)); cr4_fixed1_update(X86_CR4_UMIP, ecx, feature_bit(UMIP)); cr4_fixed1_update(X86_CR4_LA57, ecx, feature_bit(LA57)); + cr4_fixed1_update(X86_CR4_CET, ecx, feature_bit(SHSTK)); + cr4_fixed1_update(X86_CR4_CET, edx, feature_bit(IBT)); #undef cr4_fixed1_update } From patchwork Thu Jun 16 08:46:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yang, Weijiang" X-Patchwork-Id: 12883574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78AFEC43334 for ; Thu, 16 Jun 2022 08:49:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1376523AbiFPItH (ORCPT ); Thu, 16 Jun 2022 04:49:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1376395AbiFPIsy (ORCPT ); Thu, 16 Jun 2022 04:48:54 -0400 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 518545DE75; Thu, 16 Jun 2022 01:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655369269; x=1686905269; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7f++4irz+MftuwUCaj4ppD4AhCBg8dypPeUQuLNrFYQ=; b=CZPdHNG2MLrnH4Cz1KXviKnv7TKGd+WzAv/Rxf3i5fXiZ31YbuZhmq3k ZffYkWwt2sGq/bmz/av4RTqnO+OPfMrHF1iPnVwrNAqVgb7ILj+BKWMMj 2DjiRq0KJ6pbiGWa4/q1DR1LwI35tuUR0Jm/pLO/XZ0152Lqd+4lfQGl4 q3/9I9thHihe4l6jzvdpHoHmBuAlHJMg9cDUVl32u4MogQIvQcoBnmoFA 42oXuBA1g3+lQwO34M6JwUEEvwruX1H8Xrtg2zHDKQVkAFGhO9C5qChTp sFr4xVPRnhnDLiI3AP7geiGsf2Y9qzViMnW89u3t8WPpLwOFJIBHFIblT g==; X-IronPort-AV: E=McAfee;i="6400,9594,10379"; a="259055235" X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="259055235" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:42 -0700 X-IronPort-AV: E=Sophos;i="5.91,304,1647327600"; d="scan'208";a="613083184" Received: from embargo.jf.intel.com ([10.165.9.183]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Jun 2022 01:47:40 -0700 From: Yang Weijiang To: pbonzini@redhat.com, seanjc@google.com, x86@kernel.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, rick.p.edgecombe@intel.com Cc: weijiang.yang@intel.com Subject: [PATCH 19/19] KVM: x86: Enable supervisor IBT support for guest Date: Thu, 16 Jun 2022 04:46:43 -0400 Message-Id: <20220616084643.19564-20-weijiang.yang@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20220616084643.19564-1-weijiang.yang@intel.com> References: <20220616084643.19564-1-weijiang.yang@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Mainline kernel now supports supervisor IBT for kernel code, to make s-IBT work in guest(nested guest), pass through MSR_IA32_S_CET to guest(nested guest) if host kernel and KVM enabled IBT. Note, s-IBT can work independent to host xsaves support because guest MSR_IA32_S_CET can be stored/loaded from specific VMCS field. Signed-off-by: Yang Weijiang --- arch/x86/kvm/cpuid.h | 5 +++++ arch/x86/kvm/vmx/nested.c | 3 +++ arch/x86/kvm/vmx/vmx.c | 27 ++++++++++++++++++++++++--- arch/x86/kvm/x86.c | 13 ++++++++++++- 4 files changed, 44 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h index ac72aabba981..c67c1e2fc11a 100644 --- a/arch/x86/kvm/cpuid.h +++ b/arch/x86/kvm/cpuid.h @@ -230,4 +230,9 @@ static __always_inline bool guest_pv_has(struct kvm_vcpu *vcpu, return vcpu->arch.pv_cpuid.features & (1u << kvm_feature); } +static __always_inline bool cet_kernel_ibt_supported(void) +{ + return HAS_KERNEL_IBT && kvm_cpu_cap_has(X86_FEATURE_IBT); +} + #endif diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index f31f3d394507..d394136891d0 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -688,6 +688,9 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu, nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0, MSR_IA32_U_CET, MSR_TYPE_RW); + nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0, + MSR_IA32_S_CET, MSR_TYPE_RW); + nested_vmx_set_intercept_for_msr(vmx, msr_bitmap_l1, msr_bitmap_l0, MSR_IA32_PL3_SSP, MSR_TYPE_RW); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 00782d1750a5..6e7e596c0147 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -585,6 +585,7 @@ static bool is_valid_passthrough_msr(u32 msr) return true; case MSR_IA32_U_CET: case MSR_IA32_PL3_SSP: + case MSR_IA32_S_CET: return true; } @@ -1773,7 +1774,8 @@ static int vmx_get_msr_feature(struct kvm_msr_entry *msr) static bool cet_is_msr_accessible(struct kvm_vcpu *vcpu, struct msr_data *msr) { - if (!kvm_cet_user_supported()) + if (!kvm_cet_user_supported() && + !cet_kernel_ibt_supported()) return false; if (msr->host_initiated) @@ -1783,6 +1785,10 @@ static bool cet_is_msr_accessible(struct kvm_vcpu *vcpu, !guest_cpuid_has(vcpu, X86_FEATURE_IBT)) return false; + if (msr->index == MSR_IA32_S_CET && + guest_cpuid_has(vcpu, X86_FEATURE_IBT)) + return true; + if ((msr->index == MSR_IA32_PL3_SSP || msr->index == MSR_KVM_GUEST_SSP) && !guest_cpuid_has(vcpu, X86_FEATURE_SHSTK)) @@ -1933,10 +1939,13 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) case MSR_IA32_U_CET: case MSR_IA32_PL3_SSP: case MSR_KVM_GUEST_SSP: + case MSR_IA32_S_CET: if (!cet_is_msr_accessible(vcpu, msr_info)) return 1; if (msr_info->index == MSR_KVM_GUEST_SSP) msr_info->data = vmcs_readl(GUEST_SSP); + else if (msr_info->index == MSR_IA32_S_CET) + msr_info->data = vmcs_readl(GUEST_S_CET); else kvm_get_xsave_msr(msr_info); break; @@ -2273,12 +2282,16 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) vmx->pt_desc.guest.addr_a[index / 2] = data; break; case MSR_IA32_U_CET: + case MSR_IA32_S_CET: if (!cet_is_msr_accessible(vcpu, msr_info)) return 1; if ((data & GENMASK(9, 6)) || is_noncanonical_address(data, vcpu)) return 1; - kvm_set_xsave_msr(msr_info); + if (msr_index == MSR_IA32_S_CET) + vmcs_writel(GUEST_S_CET, data); + else + kvm_set_xsave_msr(msr_info); break; case MSR_IA32_PL3_SSP: case MSR_KVM_GUEST_SSP: @@ -7615,6 +7628,9 @@ static void vmx_update_intercept_for_cet_msr(struct kvm_vcpu *vcpu) incpt |= !guest_cpuid_has(vcpu, X86_FEATURE_SHSTK); vmx_set_intercept_for_msr(vcpu, MSR_IA32_PL3_SSP, MSR_TYPE_RW, incpt); + + incpt |= !guest_cpuid_has(vcpu, X86_FEATURE_IBT); + vmx_set_intercept_for_msr(vcpu, MSR_IA32_S_CET, MSR_TYPE_RW, incpt); } static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) @@ -7680,7 +7696,7 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) /* Refresh #PF interception to account for MAXPHYADDR changes. */ vmx_update_exception_bitmap(vcpu); - if (kvm_cet_user_supported()) + if (kvm_cet_user_supported() || cet_kernel_ibt_supported()) vmx_update_intercept_for_cet_msr(vcpu); } @@ -7743,6 +7759,11 @@ static __init void vmx_set_cpu_caps(void) kvm_cpu_cap_clear(X86_FEATURE_SHSTK); #endif +#ifndef CONFIG_X86_KERNEL_IBT + if (boot_cpu_has(X86_FEATURE_IBT)) + kvm_cpu_cap_clear(X86_FEATURE_IBT); +#endif + } static void vmx_request_immediate_exit(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index fe049d0e5ecc..c0118b33806a 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1463,6 +1463,7 @@ static const u32 msrs_to_save_all[] = { MSR_IA32_XFD, MSR_IA32_XFD_ERR, MSR_IA32_XSS, MSR_IA32_U_CET, MSR_IA32_PL3_SSP, MSR_KVM_GUEST_SSP, + MSR_IA32_S_CET, }; static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_all)]; @@ -6823,6 +6824,10 @@ static void kvm_init_msr_list(void) if (!kvm_cet_user_supported()) continue; break; + case MSR_IA32_S_CET: + if (!cet_kernel_ibt_supported()) + continue; + break; default: break; } @@ -11830,7 +11835,13 @@ int kvm_arch_hardware_setup(void *opaque) /* Update CET features now as kvm_caps.supported_xss is finalized. */ if (!kvm_cet_user_supported()) { kvm_cpu_cap_clear(X86_FEATURE_SHSTK); - kvm_cpu_cap_clear(X86_FEATURE_IBT); + /* If CET user bit is disabled due to cmdline option such as + * noxsaves, but kernel IBT is on, this means we can expose + * kernel IBT alone to guest since CET user mode msrs are not + * passed through to guest. + */ + if (!cet_kernel_ibt_supported()) + kvm_cpu_cap_clear(X86_FEATURE_IBT); } /*