From patchwork Fri Jan 10 18:40:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13935250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62434E7719D for ; Fri, 10 Jan 2025 18:41:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BDE76B0085; Fri, 10 Jan 2025 13:41:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 41E9D6B0093; Fri, 10 Jan 2025 13:41:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24A0F6B00D1; Fri, 10 Jan 2025 13:41:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id ECDB76B0085 for ; Fri, 10 Jan 2025 13:41:36 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A70EA80E40 for ; Fri, 10 Jan 2025 18:41:36 +0000 (UTC) X-FDA: 82992410592.05.050575C Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf30.hostedemail.com (Postfix) with ESMTP id AA5978000A for ; Fri, 10 Jan 2025 18:41:34 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=u5rBoeVP; spf=pass (imf30.hostedemail.com: domain of 33GmBZwgKCPciZbjlZmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=33GmBZwgKCPciZbjlZmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736534494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0rBDGQyJzQ3zE+AnCN6b+fGLld6jYpPH9L4MwLLlvpM=; b=ujQiZNHOl1P1bR/o3S+/E9wThjOlPwgEmdboBakjVvqdSnKsUmWWjcQVDTuSZvCyRxkjSp bcF2l4sDH/pg2dELj750WV1Y4UxC2T3zs7vQHmQmbL3zQubQcreGVq2oQgOlsIiJAwzFUG O2pu9tuAurnQcrZmjYcrw0yNB9FV1lA= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=u5rBoeVP; spf=pass (imf30.hostedemail.com: domain of 33GmBZwgKCPciZbjlZmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jackmanb.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=33GmBZwgKCPciZbjlZmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736534494; a=rsa-sha256; cv=none; b=rJgrxH+u2zXm/2XHEGuaYza9k1vO7NEAynktulef/zE9ckbEOk+AIX3OiIFf0jMTWb1krw +Ip2jdUHQXik82dC+b2ULEnKRiyycC/nNunM0FA+D8qwek/8gonRkG9yMtM6Fp8FrShU4X n92m2V1yIbzWcAiWKjMNTrD12Cn/xUw= Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-43625ceae52so12794495e9.0 for ; Fri, 10 Jan 2025 10:41:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736534493; x=1737139293; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0rBDGQyJzQ3zE+AnCN6b+fGLld6jYpPH9L4MwLLlvpM=; b=u5rBoeVP9gxSfiG6U+rju6PzKRm91FS+0q9jqq54Ds0iU0kgTlFRM6rWtpsuCYlXQw zD5nw8MgHOJk8JolT50FBqcWU7XYxgxl6B5DwdZJu2gv4Ycc4EspUeubDbgRD2QkBqId 4ciHc6FS94kb+v/mrNmKKytXfWQFwfe/EXjgEr/ve7GiQH4AA1CLB1yghnB+fTUsEFjE 3YFLu5jj35gYZyhPSyoLos5kVenF9xo8Z681vLP/gumgUxu8D4cQ3yJXs9Uod7XGkgLx ZZskWaw0eUaxkeAkSuD0XIKL66D/dllzBZMfps7rv4rTt9V0lrA6XdGvGZpnxGf0Uqu9 /MgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736534493; x=1737139293; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0rBDGQyJzQ3zE+AnCN6b+fGLld6jYpPH9L4MwLLlvpM=; b=EowhWj0t9JppKA9qHq+3bPeQoYX+cw/AAMYoZ1XOteIFG53gLjzaw6T0EQgusNgaS0 FL/3K4fwzI55Z2bdKE2TrvVSjVxHZQ35UqVlFXTwJx51WKNWrYNi0ys84drxoGE537kV IanKjuGuUv6iml+A3t38+TRsW/KMBi4lesIjJDOyNsjXgM21BlmVqPtXhCs9NPHAzUX4 Pe6h5R0n8MVlyNF1ghnMraUXVYodsVFrODrgsdzFZVdAWOve3KGQqvg1HIklvly8hxwV Rjv96pNM437O2aTDTmoTa7TeLIaYmqSBXl5GGLUU8npBNV+f/amdCRKRQCQRmZw/MMC4 AiDg== X-Forwarded-Encrypted: i=1; AJvYcCW+mQUoVwA7bD+VoXFWREbngF+L6h/ILSNnn9BXt2Oej2GzspNIfEDZX2xv4keSegbuv3jxbayMvA==@kvack.org X-Gm-Message-State: AOJu0Yx54Q7zGoCyZ5NbN1JqWR/9FP2z4Su/Tt/aUjWpGdivi/57oUOb 5QobEpvEkpH4qpXFC98XBcse1BPjFhLRhKrSnETT+Thny0fRH0/XC5O20EDNWr+DA5jxyyNw9ix Ylm+Qd6s5Bw== X-Google-Smtp-Source: AGHT+IGSdZ2gZJ+03Djx+zc677EB8kBOQn6jxeBFpWfFWpyM+wf7VvlfzHG8+toKsNSgpgcgyJMUn+Q1Gcn1Dw== X-Received: from wmbfc10.prod.google.com ([2002:a05:600c:524a:b0:434:e9fe:f913]) (user=jackmanb job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:3ba4:b0:431:3bf9:3ebb with SMTP id 5b1f17b1804b1-436e26f4805mr96359685e9.24.1736534492849; Fri, 10 Jan 2025 10:41:32 -0800 (PST) Date: Fri, 10 Jan 2025 18:40:47 +0000 In-Reply-To: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> Mime-Version: 1.0 References: <20250110-asi-rfc-v2-v2-0-8419288bc805@google.com> X-Mailer: b4 0.15-dev Message-ID: <20250110-asi-rfc-v2-v2-21-8419288bc805@google.com> Subject: [PATCH RFC v2 21/29] KVM: x86: asi: Restricted address space for VM execution From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Richard Henderson , Matt Turner , Vineet Gupta , Russell King , Catalin Marinas , Will Deacon , Guo Ren , Brian Cain , Huacai Chen , WANG Xuerui , Geert Uytterhoeven , Michal Simek , Thomas Bogendoerfer , Dinh Nguyen , Jonas Bonn , Stefan Kristiansson , Stafford Horne , "James E.J. Bottomley" , Helge Deller , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Naveen N Rao , Madhavan Srinivasan , Paul Walmsley , Palmer Dabbelt , Albert Ou , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Yoshinori Sato , Rich Felker , John Paul Adrian Glaubitz , "David S. Miller" , Andreas Larsson , Richard Weinberger , Anton Ivanov , Johannes Berg , Chris Zankel , Max Filippov , Arnd Bergmann , Andrew Morton , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Uladzislau Rezki , Christoph Hellwig , Masami Hiramatsu , Mathieu Desnoyers , Mike Rapoport , Arnaldo Carvalho de Melo , Namhyung Kim , Mark Rutland , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Dennis Zhou , Tejun Heo , Christoph Lameter , Sean Christopherson , Paolo Bonzini , Ard Biesheuvel , Josh Poimboeuf , Pawan Gupta Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-alpha@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-openrisc@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, kvm@vger.kernel.org, linux-efi@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: AA5978000A X-Rspamd-Server: rspam12 X-Stat-Signature: eghn5p7w1pscjtzf8xi7fjtrmnwg1t9h X-Rspam-User: X-HE-Tag: 1736534494-34698 X-HE-Meta: U2FsdGVkX1+3yk8nJZflxSSK+9kX3nGyyqBm/QnhT/GNMNM8Zqk8X7TUGeHwXDhP+u0ZHLM9fV6FXvT7ILD86n6zcwOhC5Q+S+mE0wJ8NRWE0zHyyRqAse/eYTuY5x6iLWovD0Q+XnS+CyezO/Qwco6x+AhhPLU25b/MKgEkt/AIKAoC1M8DsuWSHGWJLdXwkNfTyYUQDVqtHGbaxgTnLRJ7LuZcxKRtmfXyvQNgHCwbWOEiYbA5uWCkNFTRzY4GS/dRMqCNIh6/yRaa1UEOPaik+mY8SBHofBTVOarZYTA26tZy0cBV6q26fsEnkhTNxoaJYKnj7Ktc9NBzlnJnVF3XSGYNbzwyYcPCHKIysGk5ermcRjvmiax/tUq4iqbbGS71MdAFHSH4AxH5LiTF0et35SDu20YgxLS8T0iGD/A2AsEzUo/pOCuMmsu2tzjurxamXFzWW0CcpeHX1uNXLQOmbX5ZbsAgn2XqadCyd0mg89XnwLyRTJ9o03b3ecHulUGqz9uEu96Q9jCdk+4jvB5jtrfIg+QFyKEaD8oDQhQ06ttVJwH/+41JO9yS03giIpKxsrg/DXfH952hIev9Sw+A+R4bGRo7lzy2XdGrA/G+B2Fi2lIkW0nTsKpebF6IDJ5udhisWYJFsDuMx5k/DHceJzC54ZALyZRZYkkPoIgdHIVEIzOIzuyqobx2w6JFNyvSH91DpAZHklhvYg6S9LKNXran/UY603xeCebmL3dIRePpHhbltCXQz7iGhLLmaQ27EhCuTe6dXXWIEmJWTlrYGX35x9OPA9UEphutf2yMA6V9x63zgM+N043ilmWAef6IDkT4WVkh8k88vdK/ylc3mHRsPnX1ak/FEmycUKqjE9of8wmFQPtI2gbtVk3STFu5FwhXh54AKKCCKKk/E2eRL7A8PT0sfpMlwSCKSgcJRcEsgTCv28HPY+aKcPKOj4vslHzUeTyoMfcFZ5c tR+vTdzM wh7UYn9DCtgOLwy+ZW5Tp4eY8X75x4kIQMdJjHXsqymbI0WMrWYpdsiPTbySgh1rQCnu1RHRsQLvzY4Ut8tMAQ0/cMajzYsypfianVFc310zNS4RJV9wc6lU62jnvIwlh8AI7BlUGjwL2vh+Wx8nQT9r1kbFWKBTgPIH2nUjIKkWVFlr5eO7+cFsN1Q+V6esz/0cEn/KiGkY40/46CaTLtLrp8wMmGqL+FlRfAM16DiyLkgSjtTc/Ulm7q4DI5/2vLYUg/kdhwb9/bu1Nkjo/kSND8thmlLlbRWY9k5p5ef9yixE2fRpsigqByAnyXaUOL5vvRMrIgZLqlzhJe9xtWRwDKScPp4xaTRJFk+SNsKy9LXqnZqZ46o0tMPXuNzi6AgBwYzGMKZoPnvFG0e24lxEcngA9/sBhxUfX9df7S0srXJI8OZEMwbaPgRfzyvsGEwK5ALAGxTDKx9rHjYIxQfZAFMS2TkWXGgxmYwqALgBt7KepFXvFtgIqA0xAEUc7qwjIdPZJc8KFw8nktJCfBOTa1QF7VUYB1ijIeMC2I2fJvnFmQfK+opz00bg2x5mGzINK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: An ASI restricted address space is added for KVM. This protects the userspace from attack by the guest, and the guest from attack by other processes. It doesn't attempt to prevent the guest from attack by the current process. This change incorporates an extra asi_exit at the end of vcpu_run. We expect later iterations of ASI to drop that call as we gain the ability to context switch within the ASI domain. Signed-off-by: Brendan Jackman --- arch/x86/include/asm/kvm_host.h | 3 ++ arch/x86/kvm/svm/svm.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 38 ++++++++++++-------- arch/x86/kvm/x86.c | 77 ++++++++++++++++++++++++++++++++++++++++- 4 files changed, 105 insertions(+), 15 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 6d9f763a7bb9d5db422ea5625b2c28420bd14f26..00cda452dd6ca6ec57ff85ca194ee4aeb6af3be7 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -37,6 +37,7 @@ #include #include #include +#include #define __KVM_HAVE_ARCH_VCPU_DEBUGFS @@ -1535,6 +1536,8 @@ struct kvm_arch { */ #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1) struct kvm_mmu_memory_cache split_desc_cache; + + struct asi *asi; }; struct kvm_vm_stat { diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 9df3e1e5ae81a1346409632edd693cb7e0740f72..f2c3154292b4f6c960b490b0773f53bea43897bb 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4186,6 +4186,7 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in guest_state_enter_irqoff(); amd_clear_divider(); + asi_enter(vcpu->kvm->arch.asi); if (sev_es_guest(vcpu->kvm)) __svm_sev_es_vcpu_run(svm, spec_ctrl_intercepted, @@ -4193,6 +4194,7 @@ static noinstr void svm_vcpu_enter_exit(struct kvm_vcpu *vcpu, bool spec_ctrl_in else __svm_vcpu_run(svm, spec_ctrl_intercepted); + asi_relax(); guest_state_exit_irqoff(); } diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index d28618e9277ede83ad2edc1b1778ea44123aa797..181d230b1c057fed33f7b29b7b0e378dbdfeb174 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -49,6 +49,7 @@ #include #include #include +#include #include @@ -7282,14 +7283,34 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, unsigned int flags) { struct vcpu_vmx *vmx = to_vmx(vcpu); + unsigned long cr3; guest_state_enter_irqoff(); + asi_enter(vcpu->kvm->arch.asi); + + /* + * Refresh vmcs.HOST_CR3 if necessary. This must be done immediately + * prior to VM-Enter, as the kernel may load a new ASID (PCID) any time + * it switches back to the current->mm, which can occur in KVM context + * when switching to a temporary mm to patch kernel code, e.g. if KVM + * toggles a static key while handling a VM-Exit. + * Also, this must be done after asi_enter(), as it changes CR3 + * when switching address spaces. + */ + cr3 = __get_current_cr3_fast(); + if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) { + vmcs_writel(HOST_CR3, cr3); + vmx->loaded_vmcs->host_state.cr3 = cr3; + } /* * L1D Flush includes CPU buffer clear to mitigate MDS, but VERW * mitigation for MDS is done late in VMentry and is still * executed in spite of L1D Flush. This is because an extra VERW * should not matter much after the big hammer L1D Flush. + * + * This is only after asi_enter() for performance reasons. + * RFC: This also needs to be integrated with ASI's tainting model. */ if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); @@ -7310,6 +7331,8 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, vmx->idt_vectoring_info = 0; + asi_relax(); + vmx_enable_fb_clear(vmx); if (unlikely(vmx->fail)) { @@ -7338,7 +7361,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit) { struct vcpu_vmx *vmx = to_vmx(vcpu); - unsigned long cr3, cr4; + unsigned long cr4; /* Record the guest's net vcpu time for enforced NMI injections. */ if (unlikely(!enable_vnmi && @@ -7381,19 +7404,6 @@ fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, bool force_immediate_exit) vmcs_writel(GUEST_RIP, vcpu->arch.regs[VCPU_REGS_RIP]); vcpu->arch.regs_dirty = 0; - /* - * Refresh vmcs.HOST_CR3 if necessary. This must be done immediately - * prior to VM-Enter, as the kernel may load a new ASID (PCID) any time - * it switches back to the current->mm, which can occur in KVM context - * when switching to a temporary mm to patch kernel code, e.g. if KVM - * toggles a static key while handling a VM-Exit. - */ - cr3 = __get_current_cr3_fast(); - if (unlikely(cr3 != vmx->loaded_vmcs->host_state.cr3)) { - vmcs_writel(HOST_CR3, cr3); - vmx->loaded_vmcs->host_state.cr3 = cr3; - } - cr4 = cr4_read_shadow(); if (unlikely(cr4 != vmx->loaded_vmcs->host_state.cr4)) { vmcs_writel(HOST_CR4, cr4); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 83fe0a78146fc198115aba0e76ba57ecfb1dd8d9..3e0811eb510650abc601e4adce1ce4189835a730 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -85,6 +85,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include "trace.h" @@ -9674,6 +9675,55 @@ static void kvm_x86_check_cpu_compat(void *ret) *(int *)ret = kvm_x86_check_processor_compatibility(); } +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION +static inline int kvm_x86_init_asi_class(void) +{ + static struct asi_taint_policy policy = { + /* + * Prevent going to the guest with sensitive data potentially + * left in sidechannels by code running in the unrestricted + * address space, or another MM. + */ + .protect_data = ASI_TAINT_KERNEL_DATA | ASI_TAINT_OTHER_MM_DATA, + /* + * Prevent going to the guest with branch predictor state + * influenced by other processes. Note this bit is about + * protecting the guest from other parts of the system, while + * data_taints is about protecting other parts of the system + * from the guest. + */ + .prevent_control = ASI_TAINT_OTHER_MM_CONTROL, + .set = ASI_TAINT_GUEST_DATA, + }; + + /* + * Inform ASI that the guest will gain control of the branch predictor, + * unless we're just unconditionally blasting it after VM Exit. + * + * RFC: This is a bit simplified - on some configurations we could avoid + * a duplicated RSB-fill if we had a separate taint specifically for the + * RSB. + */ + if (!cpu_feature_enabled(X86_FEATURE_IBPB_ON_VMEXIT) || + !IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) || + !cpu_feature_enabled(X86_FEATURE_RSB_VMEXIT)) + policy.set = ASI_TAINT_GUEST_CONTROL; + + /* + * And the same for data left behind by code in the userspace domain + * (i.e. the VMM itself, plus kernel code serving its syscalls etc). + * This should eventually be configurable: users whose VMMs contain + * no secrets can disable it to avoid paying a mitigation cost on + * transition between their guest and userspace. + */ + policy.protect_data |= ASI_TAINT_USER_DATA; + + return asi_init_class(ASI_CLASS_KVM, &policy); +} +#else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ +static inline int kvm_x86_init_asi_class(void) { return 0; } +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) { u64 host_pat; @@ -9737,6 +9787,10 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) kvm_caps.supported_vm_types = BIT(KVM_X86_DEFAULT_VM); kvm_caps.supported_mce_cap = MCG_CTL_P | MCG_SER_P; + r = kvm_x86_init_asi_class(); + if (r < 0) + goto out_mmu_exit; + if (boot_cpu_has(X86_FEATURE_XSAVE)) { kvm_host.xcr0 = xgetbv(XCR_XFEATURE_ENABLED_MASK); kvm_caps.supported_xcr0 = kvm_host.xcr0 & KVM_SUPPORTED_XCR0; @@ -9754,7 +9808,7 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) r = ops->hardware_setup(); if (r != 0) - goto out_mmu_exit; + goto out_asi_uninit; kvm_ops_update(ops); @@ -9810,6 +9864,8 @@ int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops) out_unwind_ops: kvm_x86_ops.enable_virtualization_cpu = NULL; kvm_x86_call(hardware_unsetup)(); +out_asi_uninit: + asi_uninit_class(ASI_CLASS_KVM); out_mmu_exit: kvm_mmu_vendor_module_exit(); out_free_percpu: @@ -9841,6 +9897,7 @@ void kvm_x86_vendor_exit(void) cancel_work_sync(&pvclock_gtod_work); #endif kvm_x86_call(hardware_unsetup)(); + asi_uninit_class(ASI_CLASS_KVM); kvm_mmu_vendor_module_exit(); free_percpu(user_return_msrs); kmem_cache_destroy(x86_emulator_cache); @@ -11574,6 +11631,13 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) r = vcpu_run(vcpu); + /* + * At present ASI doesn't have the capability to transition directly + * from the restricted address space to the user address space. So we + * just return to the unrestricted address space in between. + */ + asi_exit(); + out: kvm_put_guest_fpu(vcpu); if (kvm_run->kvm_valid_regs) @@ -12705,6 +12769,14 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) if (ret) goto out_uninit_mmu; + ret = asi_init(kvm->mm, ASI_CLASS_KVM, &kvm->arch.asi); + if (ret) + goto out_uninit_mmu; + + ret = static_call(kvm_x86_vm_init)(kvm); + if (ret) + goto out_asi_destroy; + INIT_HLIST_HEAD(&kvm->arch.mask_notifier_list); atomic_set(&kvm->arch.noncoherent_dma_count, 0); @@ -12742,6 +12814,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) return 0; +out_asi_destroy: + asi_destroy(kvm->arch.asi); out_uninit_mmu: kvm_mmu_uninit_vm(kvm); kvm_page_track_cleanup(kvm); @@ -12883,6 +12957,7 @@ void kvm_arch_destroy_vm(struct kvm *kvm) kvm_destroy_vcpus(kvm); kvfree(rcu_dereference_check(kvm->arch.apic_map, 1)); kfree(srcu_dereference_check(kvm->arch.pmu_event_filter, &kvm->srcu, 1)); + asi_destroy(kvm->arch.asi); kvm_mmu_uninit_vm(kvm); kvm_page_track_cleanup(kvm); kvm_xen_destroy_vm(kvm);