From patchwork Thu Jun 1 14:23:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zeng Guang X-Patchwork-Id: 13264067 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B63AC77B7E for ; Thu, 1 Jun 2023 15:05:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233642AbjFAPFm (ORCPT ); Thu, 1 Jun 2023 11:05:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234492AbjFAPEv (ORCPT ); Thu, 1 Jun 2023 11:04:51 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6054519A1; Thu, 1 Jun 2023 08:03:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1685631816; x=1717167816; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=9N6ybH8Es2bCCrJEhSwD0nBm9Sy9h0sJf/DV0HQeEMk=; b=mIaJYOAiBux8Q+MLpEDrEBdBuwzNCEZUx+iOtYIC4Jy6ZdBu4C4xnJE7 5H1276cvfPCGDuK0TjK2Y0fSyLBcB8vVbWAc/LZKoeFYJVRXvDNPcCTW5 0wEWRcj3JABmt6wl5NO9tnU5XOmZ7YzkwX+dH5gSlKVDWD/lWEt020rr7 LHcZbcsWNFAbCNTGMak8vkRQOVtsLb9EiS1v+NXWqlPK7FlHogfURutmU 4q/uN7oWpSZvyR8KkD7cnq//I0otQMuDDe7bW3/lERqH3q9YGsB70Dp0N VKGi+WzyCCyhhoVsZzlmtcNCF5RSxBsBoFtL27mFVvk9o1CdYqV6r9c+n w==; X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="383853345" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="383853345" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="657828163" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="657828163" Received: from arthur-vostro-3668.sh.intel.com ([10.238.200.123]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:38 -0700 From: Zeng Guang To: Paolo Bonzini , Sean Christopherson , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , H Peter Anvin , kvm@vger.kernel.org Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Binbin Wu , Zeng Guang Subject: [PATCH v1 1/6] KVM: x86: Consolidate flags for __linearize() Date: Thu, 1 Jun 2023 22:23:04 +0800 Message-Id: <20230601142309.6307-2-guang.zeng@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230601142309.6307-1-guang.zeng@intel.com> References: <20230601142309.6307-1-guang.zeng@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Binbin Wu Define a 32-bit parameter and consolidate the two bools into it. __linearize() has two bool parameters write and fetch. And new flag will be needed to support new feature (e.g. LAM needs a flag to skip address untag under some conditions). No functional change intended. Signed-off-by: Binbin Wu Signed-off-by: Zeng Guang --- arch/x86/kvm/emulate.c | 19 +++++++++++++------ arch/x86/kvm/kvm_emulate.h | 4 ++++ 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index 936a397a08cd..9508836e8a35 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -687,8 +687,8 @@ static unsigned insn_alignment(struct x86_emulate_ctxt *ctxt, unsigned size) static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt, struct segmented_address addr, unsigned *max_size, unsigned size, - bool write, bool fetch, - enum x86emul_mode mode, ulong *linear) + u32 flags, enum x86emul_mode mode, + ulong *linear) { struct desc_struct desc; bool usable; @@ -696,6 +696,8 @@ static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt, u32 lim; u16 sel; u8 va_bits; + bool fetch = !!(flags & X86EMUL_F_FETCH); + bool write = !!(flags & X86EMUL_F_WRITE); la = seg_base(ctxt, addr.seg) + addr.ea; *max_size = 0; @@ -757,7 +759,11 @@ static int linearize(struct x86_emulate_ctxt *ctxt, ulong *linear) { unsigned max_size; - return __linearize(ctxt, addr, &max_size, size, write, false, + u32 flags = 0; + + if (write) + flags |= X86EMUL_F_WRITE; + return __linearize(ctxt, addr, &max_size, size, flags, ctxt->mode, linear); } @@ -768,10 +774,11 @@ static inline int assign_eip(struct x86_emulate_ctxt *ctxt, ulong dst) unsigned max_size; struct segmented_address addr = { .seg = VCPU_SREG_CS, .ea = dst }; + u32 flags = X86EMUL_F_FETCH; if (ctxt->op_bytes != sizeof(unsigned long)) addr.ea = dst & ((1UL << (ctxt->op_bytes << 3)) - 1); - rc = __linearize(ctxt, addr, &max_size, 1, false, true, ctxt->mode, &linear); + rc = __linearize(ctxt, addr, &max_size, 1, flags, ctxt->mode, &linear); if (rc == X86EMUL_CONTINUE) ctxt->_eip = addr.ea; return rc; @@ -896,6 +903,7 @@ static int __do_insn_fetch_bytes(struct x86_emulate_ctxt *ctxt, int op_size) int cur_size = ctxt->fetch.end - ctxt->fetch.data; struct segmented_address addr = { .seg = VCPU_SREG_CS, .ea = ctxt->eip + cur_size }; + u32 flags = X86EMUL_F_FETCH; /* * We do not know exactly how many bytes will be needed, and @@ -907,8 +915,7 @@ static int __do_insn_fetch_bytes(struct x86_emulate_ctxt *ctxt, int op_size) * boundary check itself. Instead, we use max_size to check * against op_size. */ - rc = __linearize(ctxt, addr, &max_size, 0, false, true, ctxt->mode, - &linear); + rc = __linearize(ctxt, addr, &max_size, 0, flags, ctxt->mode, &linear); if (unlikely(rc != X86EMUL_CONTINUE)) return rc; diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h index ab65f3a47dfd..5b9ec610b2cb 100644 --- a/arch/x86/kvm/kvm_emulate.h +++ b/arch/x86/kvm/kvm_emulate.h @@ -88,6 +88,10 @@ struct x86_instruction_info { #define X86EMUL_IO_NEEDED 5 /* IO is needed to complete emulation */ #define X86EMUL_INTERCEPTED 6 /* Intercepted by nested VMCB/VMCS */ +/* x86-specific emulation flags */ +#define X86EMUL_F_FETCH BIT(0) +#define X86EMUL_F_WRITE BIT(1) + struct x86_emulate_ops { void (*vm_bugged)(struct x86_emulate_ctxt *ctxt); /* From patchwork Thu Jun 1 14:23:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zeng Guang X-Patchwork-Id: 13264069 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B649C7EE23 for ; Thu, 1 Jun 2023 15:05:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234820AbjFAPFt (ORCPT ); Thu, 1 Jun 2023 11:05:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234933AbjFAPEv (ORCPT ); Thu, 1 Jun 2023 11:04:51 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CE85FE7A; Thu, 1 Jun 2023 08:03:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1685631817; x=1717167817; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=kZ/mbZNr5jyMMC2Stb6kF5aPD2uJleqw++i6qKuFJOM=; b=eSNGS1F09TPtjl5GIWIAAReBSwjX443gFqeA2hhJF4wfOKcPYsvnZoD4 o8vSM21iBQ2rRu28tzPcf9rT1MWZyeQc6CZuJukMp8ATSLOW0lKGMdnAS Bx1NvaKiSwCCha7/FH9mdg5BNRlvQZUNA4uSl+8a1wMdUh5VkCkpz+Bps WWltyIGeF9ONobL6FMFE+Uhb0XiT+SXVcYAxznUjvt+34dK7i8bxVt1ZO 8MR5GRkRsSl7audL+uWSVkGMNq96MnQvzlQ7JZEl884c6zMnaLkbQO99L AkT2JFr7D27GbkunjQPKVV+ssTmzK7hytKQ0DPCh0tT7H9viiTV+JA+45 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="383853377" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="383853377" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="657828187" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="657828187" Received: from arthur-vostro-3668.sh.intel.com ([10.238.200.123]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:42 -0700 From: Zeng Guang To: Paolo Bonzini , Sean Christopherson , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , H Peter Anvin , kvm@vger.kernel.org Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Zeng Guang Subject: [PATCH v1 2/6] KVM: x86: Virtualize CR4.LASS Date: Thu, 1 Jun 2023 22:23:05 +0800 Message-Id: <20230601142309.6307-3-guang.zeng@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230601142309.6307-1-guang.zeng@intel.com> References: <20230601142309.6307-1-guang.zeng@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Virtualize CR4.LASS[bit 27] under KVM control instead of being guest-owned as CR4.LASS generally set once for each vCPU at boot time and won't be toggled at runtime. Besides, only if VM has LASS capability enumerated with CPUID.(EAX=07H.ECX=1):EAX.LASS[bit 6], KVM allows guest software to be able to set CR4.LASS. Updating cr4_fixed1 to set CR4.LASS bit in the emulated IA32_VMX_CR4_FIXED1 MSR for guests and allow guests to enable LASS in nested VMX operaion as well. Notes: Setting CR4.LASS to 1 enable LASS in IA-32e mode. It doesn't take effect in legacy mode even if CR4.LASS is set. Signed-off-by: Zeng Guang Tested-by: Xuelian Guo --- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.h | 2 ++ 3 files changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index fb9d1f2d6136..92d8e65fe88c 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -125,7 +125,7 @@ | X86_CR4_PGE | X86_CR4_PCE | X86_CR4_OSFXSR | X86_CR4_PCIDE \ | X86_CR4_OSXSAVE | X86_CR4_SMEP | X86_CR4_FSGSBASE \ | X86_CR4_OSXMMEXCPT | X86_CR4_LA57 | X86_CR4_VMXE \ - | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP)) + | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP | X86_CR4_LASS)) #define CR8_RESERVED_BITS (~(unsigned long)X86_CR8_TPR) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 44fb619803b8..a33205ded85c 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7603,6 +7603,9 @@ static void nested_vmx_cr_fixed1_bits_update(struct kvm_vcpu *vcpu) cr4_fixed1_update(X86_CR4_UMIP, ecx, feature_bit(UMIP)); cr4_fixed1_update(X86_CR4_LA57, ecx, feature_bit(LA57)); + entry = kvm_find_cpuid_entry_index(vcpu, 0x7, 1); + cr4_fixed1_update(X86_CR4_LASS, eax, feature_bit(LASS)); + #undef cr4_fixed1_update } diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index c544602d07a3..e1295f490308 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -529,6 +529,8 @@ bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); __reserved_bits |= X86_CR4_VMXE; \ if (!__cpu_has(__c, X86_FEATURE_PCID)) \ __reserved_bits |= X86_CR4_PCIDE; \ + if (!__cpu_has(__c, X86_FEATURE_LASS)) \ + __reserved_bits |= X86_CR4_LASS; \ __reserved_bits; \ }) From patchwork Thu Jun 1 14:23:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zeng Guang X-Patchwork-Id: 13264068 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EB0EC7EE2A for ; Thu, 1 Jun 2023 15:05:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234370AbjFAPFp (ORCPT ); Thu, 1 Jun 2023 11:05:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235127AbjFAPFG (ORCPT ); Thu, 1 Jun 2023 11:05:06 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DC1D1A6; Thu, 1 Jun 2023 08:04:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1685631849; x=1717167849; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=atIMQ453MmU200T6ZuCKJrrV46VwSuyWV8VVDSHc8gU=; b=UXHtoTY6vVugtxldXqUVAkwXaD9IOcMO0PJ7rJdmx9QnpzsmlKLDYJGK zFabKc4sPW7LYKIvOuYmXtLXkJZaoV7pHLqxg5HK+AD7kDZQZfslcwF18 wTgLq0oJm69OHZ94blnsltGIJGzAgzXT9TDFz7RBFCFIAvCZw2bkOuFY0 jTKXKZkqclAsSE0+Q151xkdjmh4OkUz2bm8nhVph6RBGVAEZkXHwQF2H7 ZzOBeY8i0oeyNFdrGh/nzxwnAtTYTmYLlRy56Kvxc6sRJ1euGosmhdnlZ Hgh/JzAg+1BQgr94d3eKP5yuFflLTia+Dr9fw+B4C5x/yuEsYoZsJyY54 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="383853415" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="383853415" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="657828205" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="657828205" Received: from arthur-vostro-3668.sh.intel.com ([10.238.200.123]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:45 -0700 From: Zeng Guang To: Paolo Bonzini , Sean Christopherson , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , H Peter Anvin , kvm@vger.kernel.org Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Zeng Guang Subject: [PATCH v1 3/6] KVM: VMX: Add new ops in kvm_x86_ops for LASS violation check Date: Thu, 1 Jun 2023 22:23:06 +0800 Message-Id: <20230601142309.6307-4-guang.zeng@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230601142309.6307-1-guang.zeng@intel.com> References: <20230601142309.6307-1-guang.zeng@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Intel introduces LASS (Linear Address Separation) feature providing an independent mechanism to achieve the mode-based protection. LASS partitions 64-bit linear address space into two halves, user-mode address (LA[bit 63]=0) and supervisor-mode address (LA[bit 63]=1). It stops any code execution or conditional data access[1] 1. from user mode to supervisor-mode address space 2. from supervisor mode to user-mode address space and generates LASS violation fault accordingly. [1]A supervisor mode data access causes a LASS violation only if supervisor mode access protection is enabled (CR4.SMAP = 1) and either RFLAGS.AC = 0 or the access implicitly accesses a system data structure. Following are the rules of LASS violation check on the linear address(LA). User access to supervisor-mode address space: LA[bit 63] && (CPL == 3) Supervisor access to user-mode address space: Instruction fetch: !LA[bit 63] && (CPL < 3) Data access: !LA[bit 63] && (CR4.SMAP==1) && ((RFLAGS.AC == 0 && CPL < 3) || Implicit supervisor access) Add new ops in kvm_x86_ops to do LASS violation check. Signed-off-by: Zeng Guang Tested-by: Xuelian Guo --- arch/x86/include/asm/kvm-x86-ops.h | 3 +- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/kvm_emulate.h | 1 + arch/x86/kvm/vmx/vmx.c | 47 ++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.h | 2 ++ 5 files changed, 54 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index 13bc212cd4bc..8980a3bfa687 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -132,7 +132,8 @@ KVM_X86_OP_OPTIONAL(migrate_timers) KVM_X86_OP(msr_filter_changed) KVM_X86_OP(complete_emulated_msr) KVM_X86_OP(vcpu_deliver_sipi_vector) -KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons); +KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons) +KVM_X86_OP_OPTIONAL_RET0(check_lass) #undef KVM_X86_OP #undef KVM_X86_OP_OPTIONAL diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 92d8e65fe88c..98666d1e7727 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1731,6 +1731,8 @@ struct kvm_x86_ops { * Returns vCPU specific APICv inhibit reasons */ unsigned long (*vcpu_get_apicv_inhibit_reasons)(struct kvm_vcpu *vcpu); + + bool (*check_lass)(struct kvm_vcpu *vcpu, u64 access, u64 la, u32 flags); }; struct kvm_x86_nested_ops { diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h index 5b9ec610b2cb..f1439ab7c14b 100644 --- a/arch/x86/kvm/kvm_emulate.h +++ b/arch/x86/kvm/kvm_emulate.h @@ -91,6 +91,7 @@ struct x86_instruction_info { /* x86-specific emulation flags */ #define X86EMUL_F_FETCH BIT(0) #define X86EMUL_F_WRITE BIT(1) +#define X86EMUL_F_SKIPLASS BIT(2) struct x86_emulate_ops { void (*vm_bugged)(struct x86_emulate_ctxt *ctxt); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index a33205ded85c..876997e8448e 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -8130,6 +8130,51 @@ static void vmx_vm_destroy(struct kvm *kvm) free_pages((unsigned long)kvm_vmx->pid_table, vmx_get_pid_table_order(kvm)); } +/* + * Determine whether an access to the linear address causes a LASS violation. + * LASS protection is only effective in long mode. As a prerequisite, caller + * should make sure vCPU running in long mode and invoke this api to do LASS + * violation check. + */ +bool vmx_check_lass(struct kvm_vcpu *vcpu, u64 access, u64 la, u32 flags) +{ + bool user_mode, user_as, rflags_ac; + + if (!!(flags & X86EMUL_F_SKIPLASS) || + !kvm_is_cr4_bit_set(vcpu, X86_CR4_LASS)) + return false; + + WARN_ON_ONCE(!is_long_mode(vcpu)); + + user_as = !(la >> 63); + + /* + * An access is a supervisor-mode access if CPL < 3 or if it implicitly + * accesses a system data structure. For implicit accesses to system + * data structure, the processor acts as if RFLAGS.AC is clear. + */ + if (access & PFERR_IMPLICIT_ACCESS) { + user_mode = false; + rflags_ac = false; + } else { + user_mode = vmx_get_cpl(vcpu) == 3; + if (!user_mode) + rflags_ac = !!(kvm_get_rflags(vcpu) & X86_EFLAGS_AC); + } + + if (user_mode == user_as) + return false; + + /* + * Supervisor-mode _data_ accesses to user address space + * cause LASS violations only if SMAP is enabled. + */ + if (!user_mode && !(access & PFERR_FETCH_MASK)) + return kvm_is_cr4_bit_set(vcpu, X86_CR4_SMAP) && !rflags_ac; + + return true; +} + static struct kvm_x86_ops vmx_x86_ops __initdata = { .name = KBUILD_MODNAME, @@ -8269,6 +8314,8 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .complete_emulated_msr = kvm_complete_insn_gp, .vcpu_deliver_sipi_vector = kvm_vcpu_deliver_sipi_vector, + + .check_lass = vmx_check_lass, }; static unsigned int vmx_handle_intel_pt_intr(void) diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 9e66531861cf..f2e775b9849b 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -433,6 +433,8 @@ void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type); u64 vmx_get_l2_tsc_offset(struct kvm_vcpu *vcpu); u64 vmx_get_l2_tsc_multiplier(struct kvm_vcpu *vcpu); +bool vmx_check_lass(struct kvm_vcpu *vcpu, u64 access, u64 la, u32 flags); + static inline void vmx_set_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type, bool value) { From patchwork Thu Jun 1 14:23:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zeng Guang X-Patchwork-Id: 13264072 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAAC9C7EE2F for ; Thu, 1 Jun 2023 15:05:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234904AbjFAPF5 (ORCPT ); Thu, 1 Jun 2023 11:05:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42418 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235125AbjFAPFG (ORCPT ); Thu, 1 Jun 2023 11:05:06 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C2EA1A8; Thu, 1 Jun 2023 08:04:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1685631851; x=1717167851; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Zmsunkuwxh88Hrkvr7oFtCCzvzM6LSfleDwmcLdBVmk=; b=JhTE1yv2UJB5Fhx+TWtrhodjF1CBniJDd/XOVkgqQmvPQnznlNmXTEZg mAVrkKZ7qsepfVLqcAdSZj90ZN5BBwJf7suxuCXS+HDMlyFkIKMqNi6df yA5NcWBUZPtwq77QaJYcC08i0VCSaMibT99htR/M4NfXIoPmz9tj/Sbch qxKSmjS9wcdIh+hoXBGkA+9a9tf5Y3mlbYV1ZpZTrGAHNSMFj67sMhFMT 3sE6IAK2O6T5Gp9MyLuozmzBHRvB9hy8CyHj2fdjoDnRowxYeMwnlEpLn DUZSI1dDhl6+IetuXby3woXxBZphj/5SOrOQHwb/Sg3vdP9Fcir7fGa8A A==; X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="383853441" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="383853441" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:51 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="657828231" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="657828231" Received: from arthur-vostro-3668.sh.intel.com ([10.238.200.123]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:48 -0700 From: Zeng Guang To: Paolo Bonzini , Sean Christopherson , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , H Peter Anvin , kvm@vger.kernel.org Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Zeng Guang Subject: [PATCH v1 4/6] KVM: x86: Add emulator helper for LASS violation check Date: Thu, 1 Jun 2023 22:23:07 +0800 Message-Id: <20230601142309.6307-5-guang.zeng@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230601142309.6307-1-guang.zeng@intel.com> References: <20230601142309.6307-1-guang.zeng@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When LASS is enabled, KVM need apply LASS violation check to instruction emulations. Add helper for the usage of x86 emulator to perform LASS protection. Signed-off-by: Zeng Guang Tested-by: Xuelian Guo --- arch/x86/kvm/kvm_emulate.h | 1 + arch/x86/kvm/x86.c | 12 ++++++++++++ 2 files changed, 13 insertions(+) diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h index f1439ab7c14b..fd1c2b22867e 100644 --- a/arch/x86/kvm/kvm_emulate.h +++ b/arch/x86/kvm/kvm_emulate.h @@ -230,6 +230,7 @@ struct x86_emulate_ops { int (*leave_smm)(struct x86_emulate_ctxt *ctxt); void (*triple_fault)(struct x86_emulate_ctxt *ctxt); int (*set_xcr)(struct x86_emulate_ctxt *ctxt, u32 index, u64 xcr); + bool (*check_lass)(struct x86_emulate_ctxt *ctxt, u64 access, u64 la, u32 flags); }; /* Type, address-of, and value of an instruction's operand. */ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index c0778ca39650..faf01fecc4ca 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8287,6 +8287,17 @@ static void emulator_vm_bugged(struct x86_emulate_ctxt *ctxt) kvm_vm_bugged(kvm); } +static bool emulator_check_lass(struct x86_emulate_ctxt *ctxt, + u64 access, u64 la, u32 flags) +{ + struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); + + if (!is_long_mode(vcpu)) + return false; + + return static_call(kvm_x86_check_lass)(vcpu, access, la, flags); +} + static const struct x86_emulate_ops emulate_ops = { .vm_bugged = emulator_vm_bugged, .read_gpr = emulator_read_gpr, @@ -8332,6 +8343,7 @@ static const struct x86_emulate_ops emulate_ops = { .leave_smm = emulator_leave_smm, .triple_fault = emulator_triple_fault, .set_xcr = emulator_set_xcr, + .check_lass = emulator_check_lass, }; static void toggle_interruptibility(struct kvm_vcpu *vcpu, u32 mask) From patchwork Thu Jun 1 14:23:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zeng Guang X-Patchwork-Id: 13264071 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3902EC7EE23 for ; Thu, 1 Jun 2023 15:05:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234870AbjFAPFz (ORCPT ); Thu, 1 Jun 2023 11:05:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235146AbjFAPFI (ORCPT ); Thu, 1 Jun 2023 11:05:08 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00EF51B4; Thu, 1 Jun 2023 08:04:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1685631852; x=1717167852; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=nkbSB1DqlVzA+xpxNo2W+YNwCLxOpYOI0OFu6SLFydQ=; b=b0ZuWlF8w5KCkq7mD60MOe+E9CdBnbTyPFqr+o6itALECBhv89e/iMKU NDUnk4BXLYmGK6g4O4w0A5l4RkJdpo1uQtSx5kHQV9dR/QAs8k1ZfOOYI +dGrfAHejz8Tq8gcniuONm25Qy69qGfuqOByOXayr9TC1sgm4n4jYIhUp 9/Jl6hwbXXiNuxeLZWlU8RAS2v5+M9ipoWZOIXvU/posPpbqDQhEH2ryS Oc93onZ5T917N13QXu8STUu8XLyMlP5Gt511ymUhVCeINkq8v8W+5uyth 2xhSqz4OaQAZIFEVCQiErnk32tKqALZgOsf1PYGaSxTP4cwg1/shMkTQD A==; X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="383853468" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="383853468" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="657828251" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="657828251" Received: from arthur-vostro-3668.sh.intel.com ([10.238.200.123]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:51 -0700 From: Zeng Guang To: Paolo Bonzini , Sean Christopherson , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , H Peter Anvin , kvm@vger.kernel.org Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Zeng Guang Subject: [PATCH v1 5/6] KVM: x86: LASS protection on KVM emulation Date: Thu, 1 Jun 2023 22:23:08 +0800 Message-Id: <20230601142309.6307-6-guang.zeng@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230601142309.6307-1-guang.zeng@intel.com> References: <20230601142309.6307-1-guang.zeng@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Do LASS violation check for instructions emulated by KVM. Note that for instructions executed in the guest directly, hardware will perform the check. Not all instruction emulation leads to accesses to guest linear addresses because 1) some instructions like CPUID, RDMSR, don't take memory as operands 2) instruction fetch in most cases is already done inside the guest. Four cases in which KVM uses a linear address to access guest memory: - KVM emulates instruction fetches or data accesses - KVM emulates implicit data access to a system data structure - VMX instruction emulation - SGX ENCLS instruction emulation LASS violation check applies to these linear addresses so as to enforce mode-based protections as hardware behaves. As exceptions, the target memory address of emulation of invlpg, branch and call instructions doesn't require LASS violation check. Signed-off-by: Zeng Guang Tested-by: Xuelian Guo --- arch/x86/kvm/emulate.c | 30 ++++++++++++++++++++++++++++-- arch/x86/kvm/vmx/nested.c | 3 +++ arch/x86/kvm/vmx/sgx.c | 4 ++++ 3 files changed, 35 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index 9508836e8a35..ed5191fa2079 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -698,6 +698,7 @@ static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt, u8 va_bits; bool fetch = !!(flags & X86EMUL_F_FETCH); bool write = !!(flags & X86EMUL_F_WRITE); + u64 access = fetch ? PFERR_FETCH_MASK : 0; la = seg_base(ctxt, addr.seg) + addr.ea; *max_size = 0; @@ -743,6 +744,10 @@ static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt, } break; } + + if (ctxt->ops->check_lass(ctxt, access, *linear, flags)) + goto bad; + if (la & (insn_alignment(ctxt, size) - 1)) return emulate_gp(ctxt, 0); return X86EMUL_CONTINUE; @@ -774,7 +779,11 @@ static inline int assign_eip(struct x86_emulate_ctxt *ctxt, ulong dst) unsigned max_size; struct segmented_address addr = { .seg = VCPU_SREG_CS, .ea = dst }; - u32 flags = X86EMUL_F_FETCH; + /* + * LASS doesn't apply to addresses that specify the targets of jump and + * call instructions. + */ + u32 flags = X86EMUL_F_FETCH | X86EMUL_F_SKIPLASS; if (ctxt->op_bytes != sizeof(unsigned long)) addr.ea = dst & ((1UL << (ctxt->op_bytes << 3)) - 1); @@ -853,6 +862,13 @@ static inline int jmp_rel(struct x86_emulate_ctxt *ctxt, int rel) static int linear_read_system(struct x86_emulate_ctxt *ctxt, ulong linear, void *data, unsigned size) { + if (ctxt->ops->check_lass(ctxt, PFERR_IMPLICIT_ACCESS, linear, 0)) { + ctxt->exception.vector = GP_VECTOR; + ctxt->exception.error_code = 0; + ctxt->exception.error_code_valid = true; + return X86EMUL_PROPAGATE_FAULT; + } + return ctxt->ops->read_std(ctxt, linear, data, size, &ctxt->exception, true); } @@ -860,6 +876,13 @@ static int linear_write_system(struct x86_emulate_ctxt *ctxt, ulong linear, void *data, unsigned int size) { + if (ctxt->ops->check_lass(ctxt, PFERR_IMPLICIT_ACCESS, linear, 0)) { + ctxt->exception.vector = GP_VECTOR; + ctxt->exception.error_code = 0; + ctxt->exception.error_code_valid = true; + return X86EMUL_PROPAGATE_FAULT; + } + return ctxt->ops->write_std(ctxt, linear, data, size, &ctxt->exception, true); } @@ -3448,8 +3471,11 @@ static int em_invlpg(struct x86_emulate_ctxt *ctxt) { int rc; ulong linear; + unsigned max_size; - rc = linearize(ctxt, ctxt->src.addr.mem, 1, false, &linear); + /* LASS doesn't apply to the memory address for invlpg */ + rc = __linearize(ctxt, ctxt->src.addr.mem, &max_size, 1, + X86EMUL_F_SKIPLASS, ctxt->mode, &linear); if (rc == X86EMUL_CONTINUE) ctxt->ops->invlpg(ctxt, linear); /* Disable writeback. */ diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index e35cf0bd0df9..bb1c3fa13c13 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -4986,6 +4986,9 @@ int get_vmx_mem_address(struct kvm_vcpu *vcpu, unsigned long exit_qualification, * destination for long mode! */ exn = is_noncanonical_address(*ret, vcpu); + + if (!exn) + exn = vmx_check_lass(vcpu, 0, *ret, 0); } else { /* * When not in long mode, the virtual/linear address is diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c index 2261b684a7d4..3825275827eb 100644 --- a/arch/x86/kvm/vmx/sgx.c +++ b/arch/x86/kvm/vmx/sgx.c @@ -46,6 +46,10 @@ static int sgx_get_encls_gva(struct kvm_vcpu *vcpu, unsigned long offset, ((s.base != 0 || s.limit != 0xffffffff) && (((u64)*gva + size - 1) > s.limit + 1)); } + + if (!fault && is_long_mode(vcpu)) + fault = vmx_check_lass(vcpu, 0, *gva, 0); + if (fault) kvm_inject_gp(vcpu, 0); return fault ? -EINVAL : 0; From patchwork Thu Jun 1 14:23:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zeng Guang X-Patchwork-Id: 13264070 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEAC7C7EE2A for ; Thu, 1 Jun 2023 15:05:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234853AbjFAPFy (ORCPT ); Thu, 1 Jun 2023 11:05:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235334AbjFAPF2 (ORCPT ); Thu, 1 Jun 2023 11:05:28 -0400 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3BA810E9; Thu, 1 Jun 2023 08:04:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1685631874; x=1717167874; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=Qobfjghbe5gi6AIkAjOe1SXmoivnNVnCI+24keSLXNs=; b=ahgAhLOD/8SwtU/IAVU21c2CbVTj02Be++UwKkW0PFxjGhsvipBbRzOc xsfTu8THe0sQ+ofqkGOVuoEwI3p3uoYUuGS82WVLpmzHsdAvCWVDXzrCq 8yAZNaJl8vWGrOA/xpOOd+386aShwmnsRLsCQuwd0ymMwEeNOv0/uvkke 43vQbHw8weCxf2UInOeWM09mqSuyJ9+efP6kwVLTI+tGdIR8qBdCxFWvH P1S0zFBq0HcCZhvJPDKVH7m91EvtMfdauxdJHnMxVt8BD33D2dcLqVdIe Lm2Bs52CBnSEtsoM04GxH4rUB/I9WsLwbicctXqXR3SGBu2U8TwBprnVS A==; X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="383853504" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="383853504" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:57 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10728"; a="657828278" X-IronPort-AV: E=Sophos;i="6.00,210,1681196400"; d="scan'208";a="657828278" Received: from arthur-vostro-3668.sh.intel.com ([10.238.200.123]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Jun 2023 08:02:54 -0700 From: Zeng Guang To: Paolo Bonzini , Sean Christopherson , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , H Peter Anvin , kvm@vger.kernel.org Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Zeng Guang Subject: [PATCH v1 6/6] KVM: x86: Advertise LASS CPUID to user space Date: Thu, 1 Jun 2023 22:23:09 +0800 Message-Id: <20230601142309.6307-7-guang.zeng@intel.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230601142309.6307-1-guang.zeng@intel.com> References: <20230601142309.6307-1-guang.zeng@intel.com> Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org LASS (Linear-address space separation) is an independent mechanism to enforce the mode-based protection that can prevent user-mode accesses to supervisor-mode addresses, and vice versa. Because the LASS protections are applied before paging, malicious software can not acquire any paging-based timing information to compromise the security of system. The CPUID bit definition to support LASS: CPUID.(EAX=07H.ECX=1):EAX.LASS[bit 6] Advertise LASS to user space to support LASS virtualization. Signed-off-by: Zeng Guang Tested-by: Xuelian Guo --- arch/x86/kvm/cpuid.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 0c9660a07b23..a7fafe99ffe4 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -646,9 +646,8 @@ void kvm_set_cpu_caps(void) kvm_cpu_cap_set(X86_FEATURE_SPEC_CTRL_SSBD); kvm_cpu_cap_mask(CPUID_7_1_EAX, - F(AVX_VNNI) | F(AVX512_BF16) | F(CMPCCXADD) | - F(FZRM) | F(FSRS) | F(FSRC) | - F(AMX_FP16) | F(AVX_IFMA) + F(AVX_VNNI) | F(AVX512_BF16) | F(LASS) | F(CMPCCXADD) | + F(FZRM) | F(FSRS) | F(FSRC) | F(AMX_FP16) | F(AVX_IFMA) ); kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX,