From patchwork Mon Oct 29 15:40:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Julian Stecklina X-Patchwork-Id: 10659453 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A41E814E2 for ; Mon, 29 Oct 2018 15:41:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FB0729B58 for ; Mon, 29 Oct 2018 15:41:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D62529B67; Mon, 29 Oct 2018 15:41:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 39DCB29B58 for ; Mon, 29 Oct 2018 15:41:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727520AbeJ3AaV (ORCPT ); Mon, 29 Oct 2018 20:30:21 -0400 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:41511 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727085AbeJ3AaV (ORCPT ); Mon, 29 Oct 2018 20:30:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1540827672; x=1572363672; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=OrubCFTf0hB4Sr8BqyPYwAdKbyOGbpSlySqIX4oovJ4=; b=Hg8WBgUXKdik9lZmapUw2UcxrZ9Ma1CHVTljMWWwzII+vBYKaPOhhlA+ Jn+M/0SR6iYRQtXjMIaKp20WgbMl7THfJfrnjPVBzI7Jv9q2VqXr4jWL8 toqNJQQeN4REgwFf4ieFLKK8WKTo2KQlNvQLUaKiAOuc0aY0Lf21FC1o6 8=; X-IronPort-AV: E=Sophos;i="5.54,440,1534809600"; d="scan'208";a="370915898" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-f14f4a47.us-west-2.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 29 Oct 2018 15:41:10 +0000 Received: from u54ee758033e858cfa736.ant.amazon.com (pdx2-ws-svc-lb17-vlan3.amazon.com [10.247.140.70]) by email-inbound-relay-2a-f14f4a47.us-west-2.amazon.com (8.14.7/8.14.7) with ESMTP id w9TFf5Tw096295 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 29 Oct 2018 15:41:06 GMT Received: from u54ee758033e858cfa736.ant.amazon.com (localhost [127.0.0.1]) by u54ee758033e858cfa736.ant.amazon.com (8.15.2/8.15.2/Debian-3) with ESMTP id w9TFf45k000639; Mon, 29 Oct 2018 16:41:04 +0100 Received: (from jsteckli@localhost) by u54ee758033e858cfa736.ant.amazon.com (8.15.2/8.15.2/Submit) id w9TFf49w000636; Mon, 29 Oct 2018 16:41:04 +0100 From: Julian Stecklina To: kvm@vger.kernel.org, Paolo Bonzini Cc: Julian Stecklina , Julian Stecklina , linux-kernel@vger.kernel.org Subject: [PATCH v2 1/3] kvm, vmx: move CR2 context switch out of assembly path Date: Mon, 29 Oct 2018 16:40:42 +0100 Message-Id: <74fa3809ea293cc05d37b1449b16e08480c4ddbd.1540822350.git.jsteckli@amazon.de> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The VM entry/exit path is a giant inline assembly statement. Simplify it by doing CR2 context switching in plain C. Move CR2 restore behind IBRS clearing, so we reduce the amount of code we execute with IBRS on. Signed-off-by: Julian Stecklina Reviewed-by: Jan H. Schönherr Reviewed-by: Konrad Jan Miller Reviewed-by: Jim Mattson Reviewed-by: Sean Christopherson --- arch/x86/kvm/vmx.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index ccc6a01..a6e5a5c 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -11212,6 +11212,9 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) evmcs_rsp = static_branch_unlikely(&enable_evmcs) ? (unsigned long)¤t_evmcs->host_rsp : 0; + if (read_cr2() != vcpu->arch.cr2) + write_cr2(vcpu->arch.cr2); + if (static_branch_unlikely(&vmx_l1d_should_flush)) vmx_l1d_flush(vcpu); @@ -11231,13 +11234,6 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) "2: \n\t" __ex("vmwrite %%" _ASM_SP ", %%" _ASM_DX) "\n\t" "1: \n\t" - /* Reload cr2 if changed */ - "mov %c[cr2](%0), %%" _ASM_AX " \n\t" - "mov %%cr2, %%" _ASM_DX " \n\t" - "cmp %%" _ASM_AX ", %%" _ASM_DX " \n\t" - "je 3f \n\t" - "mov %%" _ASM_AX", %%cr2 \n\t" - "3: \n\t" /* Check if vmlaunch of vmresume is needed */ "cmpl $0, %c[launched](%0) \n\t" /* Load guest registers. Don't clobber flags. */ @@ -11298,8 +11294,6 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) "xor %%r14d, %%r14d \n\t" "xor %%r15d, %%r15d \n\t" #endif - "mov %%cr2, %%" _ASM_AX " \n\t" - "mov %%" _ASM_AX ", %c[cr2](%0) \n\t" "xor %%eax, %%eax \n\t" "xor %%ebx, %%ebx \n\t" @@ -11331,7 +11325,6 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) [r14]"i"(offsetof(struct vcpu_vmx, vcpu.arch.regs[VCPU_REGS_R14])), [r15]"i"(offsetof(struct vcpu_vmx, vcpu.arch.regs[VCPU_REGS_R15])), #endif - [cr2]"i"(offsetof(struct vcpu_vmx, vcpu.arch.cr2)), [wordsize]"i"(sizeof(ulong)) : "cc", "memory" #ifdef CONFIG_X86_64 @@ -11365,6 +11358,8 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) /* Eliminate branch target predictions from guest mode */ vmexit_fill_RSB(); + vcpu->arch.cr2 = read_cr2(); + /* All fields are clean at this point */ if (static_branch_unlikely(&enable_evmcs)) current_evmcs->hv_clean_fields |= From patchwork Mon Oct 29 15:40:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Julian Stecklina X-Patchwork-Id: 10659457 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AE4EF174F for ; Mon, 29 Oct 2018 15:41:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ABD3A29ADC for ; Mon, 29 Oct 2018 15:41:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AA36B29B5E; Mon, 29 Oct 2018 15:41:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4341529ADC for ; Mon, 29 Oct 2018 15:41:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727629AbeJ3Aab (ORCPT ); Mon, 29 Oct 2018 20:30:31 -0400 Received: from smtp-fw-6001.amazon.com ([52.95.48.154]:64039 "EHLO smtp-fw-6001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727085AbeJ3Aab (ORCPT ); Mon, 29 Oct 2018 20:30:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1540827682; x=1572363682; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=BZyxgBHLGH4Lg47rL08aS0lODyuUZeuDeiy+P3SGpnA=; b=e5YeeAWDpf5Bt816v8RA5LK6bM9un/9A6WyS1jekUTtowkV1pDQKtyeM 7jxyfV8YIxcsgyiPYwd9h6V9AGEvxgjrj0Frjfp/VppQe8o6sF2inQojv rLzM0eyaRDZENin4YAg22WTbYhTVbXTmkP1a7DGOwykbSU5A+KOE0Wlke k=; X-IronPort-AV: E=Sophos;i="5.54,440,1534809600"; d="scan'208";a="365610272" Received: from iad6-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-69849ee2.us-west-2.amazon.com) ([10.124.125.6]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 29 Oct 2018 15:41:20 +0000 Received: from u54ee758033e858cfa736.ant.amazon.com (pdx2-ws-svc-lb17-vlan3.amazon.com [10.247.140.70]) by email-inbound-relay-2a-69849ee2.us-west-2.amazon.com (8.14.7/8.14.7) with ESMTP id w9TFfGIp038342 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 29 Oct 2018 15:41:17 GMT Received: from u54ee758033e858cfa736.ant.amazon.com (localhost [127.0.0.1]) by u54ee758033e858cfa736.ant.amazon.com (8.15.2/8.15.2/Debian-3) with ESMTP id w9TFfFcH000667; Mon, 29 Oct 2018 16:41:15 +0100 Received: (from jsteckli@localhost) by u54ee758033e858cfa736.ant.amazon.com (8.15.2/8.15.2/Submit) id w9TFfFJH000666; Mon, 29 Oct 2018 16:41:15 +0100 From: Julian Stecklina To: kvm@vger.kernel.org, Paolo Bonzini Cc: Julian Stecklina , Julian Stecklina , linux-kernel@vger.kernel.org Subject: [PATCH v2 2/3] kvm, vmx: move register clearing out of assembly path Date: Mon, 29 Oct 2018 16:40:43 +0100 Message-Id: <33085f6c98496ed8094ce68855b49851abb86406.1540822350.git.jsteckli@amazon.de> X-Mailer: git-send-email 2.7.4 In-Reply-To: <74fa3809ea293cc05d37b1449b16e08480c4ddbd.1540822350.git.jsteckli@amazon.de> References: <74fa3809ea293cc05d37b1449b16e08480c4ddbd.1540822350.git.jsteckli@amazon.de> In-Reply-To: <74fa3809ea293cc05d37b1449b16e08480c4ddbd.1540822350.git.jsteckli@amazon.de> References: <74fa3809ea293cc05d37b1449b16e08480c4ddbd.1540822350.git.jsteckli@amazon.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Split the security related register clearing out of the large inline assembly VM entry path. This results in two slightly less complicated inline assembly statements, where it is clearer what each one does. Signed-off-by: Julian Stecklina Reviewed-by: Jan H. Schönherr Reviewed-by: Konrad Jan Miller Reviewed-by: Jim Mattson --- arch/x86/kvm/vmx.c | 45 ++++++++++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 17 deletions(-) diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index a6e5a5c..29a2ee7 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -11281,24 +11281,7 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) "mov %%r13, %c[r13](%0) \n\t" "mov %%r14, %c[r14](%0) \n\t" "mov %%r15, %c[r15](%0) \n\t" - /* - * Clear host registers marked as clobbered to prevent - * speculative use. - */ - "xor %%r8d, %%r8d \n\t" - "xor %%r9d, %%r9d \n\t" - "xor %%r10d, %%r10d \n\t" - "xor %%r11d, %%r11d \n\t" - "xor %%r12d, %%r12d \n\t" - "xor %%r13d, %%r13d \n\t" - "xor %%r14d, %%r14d \n\t" - "xor %%r15d, %%r15d \n\t" #endif - - "xor %%eax, %%eax \n\t" - "xor %%ebx, %%ebx \n\t" - "xor %%esi, %%esi \n\t" - "xor %%edi, %%edi \n\t" "pop %%" _ASM_BP "; pop %%" _ASM_DX " \n\t" ".pushsection .rodata \n\t" ".global vmx_return \n\t" @@ -11336,6 +11319,34 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu) ); /* + * Don't let guest register values survive. Registers that cannot + * contain guest values anymore are not touched. + */ + asm volatile ( + "xor %%eax, %%eax \n\t" + "xor %%ebx, %%ebx \n\t" + "xor %%esi, %%esi \n\t" + "xor %%edi, %%edi \n\t" +#ifdef CONFIG_X86_64 + "xor %%r8d, %%r8d \n\t" + "xor %%r9d, %%r9d \n\t" + "xor %%r10d, %%r10d \n\t" + "xor %%r11d, %%r11d \n\t" + "xor %%r12d, %%r12d \n\t" + "xor %%r13d, %%r13d \n\t" + "xor %%r14d, %%r14d \n\t" + "xor %%r15d, %%r15d \n\t" +#endif + ::: "cc" +#ifdef CONFIG_X86_64 + , "rax", "rbx", "rsi", "rdi" + , "r8", "r9", "r10", "r11", "r12", "r13", "r14", "r15" +#else + , "eax", "ebx", "esi", "edi" +#endif + ); + + /* * We do not use IBRS in the kernel. If this vCPU has used the * SPEC_CTRL MSR it may have left it on; save the value and * turn it off. This is much more efficient than blindly adding From patchwork Mon Oct 29 15:40:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Julian Stecklina X-Patchwork-Id: 10659459 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C48F7174F for ; Mon, 29 Oct 2018 15:41:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C1D7329B49 for ; Mon, 29 Oct 2018 15:41:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BF2F329B63; Mon, 29 Oct 2018 15:41:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D4F029B57 for ; Mon, 29 Oct 2018 15:41:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727683AbeJ3Aah (ORCPT ); Mon, 29 Oct 2018 20:30:37 -0400 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:47197 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727456AbeJ3Aag (ORCPT ); Mon, 29 Oct 2018 20:30:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.de; i=@amazon.de; q=dns/txt; s=amazon201209; t=1540827687; x=1572363687; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=SdGxs4CHf5TRhTYKYm5KecJ6+mGYsQYqjbP+I6JaFLw=; b=eQ4bUXstjUcH7c0mlWutL6Of17Ft6S68I4ntSW56t6GRpfJXSWUh43Xv 7X9R2ebdZj7ns+L72vWP9R1+zWY7VkOZ3gfh2VUuAj2promgWN5Q7YdOq COmBbHYH+CHW2ZlhNXJTFKM6nDcmSM8N3tTIniMRZo0C4pVCyof4yC7YV U=; X-IronPort-AV: E=Sophos;i="5.54,440,1534809600"; d="scan'208";a="701924541" Received: from iad6-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-2c-579b7f5b.us-west-2.amazon.com) ([10.124.125.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 29 Oct 2018 15:41:25 +0000 Received: from u54ee758033e858cfa736.ant.amazon.com (pdx2-ws-svc-lb17-vlan2.amazon.com [10.247.140.66]) by email-inbound-relay-2c-579b7f5b.us-west-2.amazon.com (8.14.7/8.14.7) with ESMTP id w9TFfK5E016030 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 29 Oct 2018 15:41:22 GMT Received: from u54ee758033e858cfa736.ant.amazon.com (localhost [127.0.0.1]) by u54ee758033e858cfa736.ant.amazon.com (8.15.2/8.15.2/Debian-3) with ESMTP id w9TFfKMr000676; Mon, 29 Oct 2018 16:41:20 +0100 Received: (from jsteckli@localhost) by u54ee758033e858cfa736.ant.amazon.com (8.15.2/8.15.2/Submit) id w9TFfJ0Q000675; Mon, 29 Oct 2018 16:41:19 +0100 From: Julian Stecklina To: kvm@vger.kernel.org, Paolo Bonzini Cc: Julian Stecklina , Julian Stecklina , linux-kernel@vger.kernel.org Subject: [PATCH v2 3/3] kvm, vmx: fix __invvpid style Date: Mon, 29 Oct 2018 16:40:44 +0100 Message-Id: X-Mailer: git-send-email 2.7.4 In-Reply-To: <74fa3809ea293cc05d37b1449b16e08480c4ddbd.1540822350.git.jsteckli@amazon.de> References: <74fa3809ea293cc05d37b1449b16e08480c4ddbd.1540822350.git.jsteckli@amazon.de> In-Reply-To: <74fa3809ea293cc05d37b1449b16e08480c4ddbd.1540822350.git.jsteckli@amazon.de> References: <74fa3809ea293cc05d37b1449b16e08480c4ddbd.1540822350.git.jsteckli@amazon.de> MIME-Version: 1.0 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The code violated the coding style. Fixed by using tabs instead of spaces. There are only whitespace changes here. Signed-off-by: Julian Stecklina Reviewed-by: Jan H. Schönherr Reviewed-by: Konrad Jan Miller --- arch/x86/kvm/vmx.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c index 29a2ee7..9bd7c6c 100644 --- a/arch/x86/kvm/vmx.c +++ b/arch/x86/kvm/vmx.c @@ -2100,16 +2100,16 @@ static int __find_msr_index(struct vcpu_vmx *vmx, u32 msr) static inline void __invvpid(unsigned long ext, u16 vpid, gva_t gva) { - struct { - u64 vpid : 16; - u64 rsvd : 48; - u64 gva; - } operand = { vpid, 0, gva }; - bool error; - - asm volatile (__ex("invvpid %2, %1") CC_SET(na) - : CC_OUT(na) (error) : "r"(ext), "m"(operand)); - BUG_ON(error); + struct { + u64 vpid : 16; + u64 rsvd : 48; + u64 gva; + } operand = { vpid, 0, gva }; + bool error; + + asm volatile (__ex("invvpid %2, %1") CC_SET(na) + : CC_OUT(na) (error) : "r"(ext), "m"(operand)); + BUG_ON(error); } static inline void __invept(unsigned long ext, u64 eptp, gpa_t gpa)