From patchwork Mon Feb 3 17:37:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 11363129 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0ED6417E0 for ; Mon, 3 Feb 2020 17:38:43 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DD9682080C for ; Mon, 3 Feb 2020 17:38:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="A/ZLO0dI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DD9682080C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iyffU-0007RY-Nh; Mon, 03 Feb 2020 17:37:52 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iyffU-0007RF-3h for xen-devel@lists.xenproject.org; Mon, 03 Feb 2020 17:37:52 +0000 X-Inumbo-ID: e623d650-46ab-11ea-b211-bc764e2007e4 Received: from esa2.hc3370-68.iphmx.com (unknown [216.71.145.153]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id e623d650-46ab-11ea-b211-bc764e2007e4; Mon, 03 Feb 2020 17:37:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1580751471; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=5MEa/49cobXk7EMoIuYalIAf1G63BAeOC6hXVgfCBaw=; b=A/ZLO0dIW8UswyM7P07nPzRlXphOPYVs4jD2F9uG+eiaXnuce+xCcrS0 4tuLGW+cx/7qPcgblB40ux+Ud61jAUq05S273dlLhRJFk3gzsV5GN2YnF OkuPrYtcokJi7Kt96weTHbUj9LfvpR2Z5oxSMlproDhhpy9l10Sl/uzZy g=; Authentication-Results: esa2.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=roger.pau@citrix.com; spf=Pass smtp.mailfrom=roger.pau@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa2.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa2.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa2.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: mHlxZMSE87ZAjY1qftg3u18/cnXfPg0ahuxrViJYhEVJOEPm9UG10ocv8BD+2TJNOOMxJmMS3E xl0rqR66CVNn4ftNCCyFEb+iyaH4eAkBkvoYP0dW0dVtsArYJQou1Fyhofuq1888FUUtnQPji+ k3Fv2JrlaRxvDHN4t2QzJ069dLAhenPO9n/8Ci87TNzd+8YttUb5HUhmJyfsrte5TaD5ujnekd LcoRK7c1710Qj6PO1G4V4zqA7pJDUlZVURIkLlnyIU9XNXATPpb/6vgeWy90a+/c9llCWKTN27 Ybo= X-SBRS: 2.7 X-MesageID: 11874324 X-Ironport-Server: esa2.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.70,398,1574139600"; d="scan'208";a="11874324" From: Roger Pau Monne To: Date: Mon, 3 Feb 2020 18:37:27 +0100 Message-ID: <20200203173728.18135-2-roger.pau@citrix.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200203173728.18135-1-roger.pau@citrix.com> References: <20200203173728.18135-1-roger.pau@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 1/2] nvmx: implement support for MSR bitmaps X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Kevin Tian , Jun Nakajima , Wei Liu , Andrew Cooper , Roger Pau Monne Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Current implementation of nested VMX has a half baked handling of MSR bitmaps for the L1 VMM: it maps the L1 VMM provided MSR bitmap, but doesn't actually load it into the nested vmcs, and thus the nested guest vmcs ends up using the same MSR bitmap as the L1 VMM. This is wrong as there's no assurance that the set of features enabled for the L1 vmcs are the same that L1 itself is going to use in the nested vmcs, and thus can lead to misconfigurations. For example L1 vmcs can use x2APIC virtualization and virtual interrupt delivery, and thus some x2APIC MSRs won't be trapped so that they can be handled directly by the hardware using virtualization extensions. On the other hand, the nested vmcs created by L1 VMM might not use any of such features, so using a MSR bitmap that doesn't trap accesses to the x2APIC MSRs will be leaking them to the underlying hardware. Fix this by crafting a merged MSR bitmap between the one used by L1 and the nested guest. Signed-off-by: Roger Pau Monné Reviewed-by: Kevin Tian --- Changes since v2: - Pass shadow_ctrl into update_msrbitmap, and check there if CPU_BASED_ACTIVATE_MSR_BITMAP is set. - Do not enable MSR bitmap unless it's enabled in both L1 and L2. - Rename L1 guest to L2 in nestedvmx struct comment. Changes since v1: - Split the x2APIC MSR fix into a separate patch. - Move setting MSR_BITMAP vmcs field into load_vvmcs_host_state for virtual vmexit. - Allocate memory with MEMF_no_owner. - Use tabs to align comment of the nestedvmx struct field. --- xen/arch/x86/hvm/vmx/vvmx.c | 62 ++++++++++++++++++++++++++++-- xen/include/asm-x86/hvm/vmx/vvmx.h | 3 +- 2 files changed, 61 insertions(+), 4 deletions(-) diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c index 47eee1e5b9..f118f88683 100644 --- a/xen/arch/x86/hvm/vmx/vvmx.c +++ b/xen/arch/x86/hvm/vmx/vvmx.c @@ -128,6 +128,16 @@ int nvmx_vcpu_initialise(struct vcpu *v) unmap_domain_page(vw); } + if ( cpu_has_vmx_msr_bitmap ) + { + nvmx->msr_merged = alloc_domheap_page(d, MEMF_no_owner); + if ( !nvmx->msr_merged ) + { + gdprintk(XENLOG_ERR, "nest: allocation for MSR bitmap failed\n"); + return -ENOMEM; + } + } + nvmx->ept.enabled = 0; nvmx->guest_vpid = 0; nvmx->vmxon_region_pa = INVALID_PADDR; @@ -182,6 +192,11 @@ void nvmx_vcpu_destroy(struct vcpu *v) free_domheap_page(v->arch.hvm.vmx.vmwrite_bitmap); v->arch.hvm.vmx.vmwrite_bitmap = NULL; } + if ( nvmx->msr_merged ) + { + free_domheap_page(nvmx->msr_merged); + nvmx->msr_merged = NULL; + } } void nvmx_domain_relinquish_resources(struct domain *d) @@ -548,6 +563,35 @@ unsigned long *_shadow_io_bitmap(struct vcpu *v) return nestedhvm_vcpu_iomap_get(port80, portED); } +static void update_msrbitmap(struct vcpu *v, uint32_t shadow_ctrl) +{ + struct nestedvmx *nvmx = &vcpu_2_nvmx(v); + struct vmx_msr_bitmap *msr_bitmap; + + if ( !(shadow_ctrl & CPU_BASED_ACTIVATE_MSR_BITMAP) || + !nvmx->msrbitmap ) + return; + + msr_bitmap = __map_domain_page(nvmx->msr_merged); + + bitmap_or(msr_bitmap->read_low, nvmx->msrbitmap->read_low, + v->arch.hvm.vmx.msr_bitmap->read_low, + sizeof(msr_bitmap->read_low) * 8); + bitmap_or(msr_bitmap->read_high, nvmx->msrbitmap->read_high, + v->arch.hvm.vmx.msr_bitmap->read_high, + sizeof(msr_bitmap->read_high) * 8); + bitmap_or(msr_bitmap->write_low, nvmx->msrbitmap->write_low, + v->arch.hvm.vmx.msr_bitmap->write_low, + sizeof(msr_bitmap->write_low) * 8); + bitmap_or(msr_bitmap->write_high, nvmx->msrbitmap->write_high, + v->arch.hvm.vmx.msr_bitmap->write_high, + sizeof(msr_bitmap->write_high) * 8); + + unmap_domain_page(msr_bitmap); + + __vmwrite(MSR_BITMAP, page_to_maddr(nvmx->msr_merged)); +} + void nvmx_update_exec_control(struct vcpu *v, u32 host_cntrl) { u32 pio_cntrl = (CPU_BASED_ACTIVATE_IO_BITMAP @@ -558,10 +602,17 @@ void nvmx_update_exec_control(struct vcpu *v, u32 host_cntrl) shadow_cntrl = __n2_exec_control(v); pio_cntrl &= shadow_cntrl; /* Enforce the removed features */ - shadow_cntrl &= ~(CPU_BASED_ACTIVATE_MSR_BITMAP - | CPU_BASED_ACTIVATE_IO_BITMAP + shadow_cntrl &= ~(CPU_BASED_ACTIVATE_IO_BITMAP | CPU_BASED_UNCOND_IO_EXITING); - shadow_cntrl |= host_cntrl; + /* + * Do NOT enforce the MSR bitmap currently used by L1, as certain hardware + * virtualization features require specific MSR bitmap settings, but + * without the guest also using these same features the bitmap could be + * leaking through unwanted MSR accesses. + */ + shadow_cntrl |= host_cntrl & ~CPU_BASED_ACTIVATE_MSR_BITMAP; + if ( !(shadow_cntrl & host_cntrl & CPU_BASED_ACTIVATE_MSR_BITMAP) ) + shadow_cntrl &= ~CPU_BASED_ACTIVATE_MSR_BITMAP; if ( pio_cntrl == CPU_BASED_UNCOND_IO_EXITING ) { /* L1 VMM intercepts all I/O instructions */ shadow_cntrl |= CPU_BASED_UNCOND_IO_EXITING; @@ -584,6 +635,8 @@ void nvmx_update_exec_control(struct vcpu *v, u32 host_cntrl) __vmwrite(IO_BITMAP_B, virt_to_maddr(bitmap) + PAGE_SIZE); } + update_msrbitmap(v, shadow_cntrl); + /* TODO: change L0 intr window to MTF or NMI window */ __vmwrite(CPU_BASED_VM_EXEC_CONTROL, shadow_cntrl); } @@ -1278,6 +1331,9 @@ static void load_vvmcs_host_state(struct vcpu *v) hvm_set_tsc_offset(v, v->arch.hvm.cache_tsc_offset, 0); set_vvmcs(v, VM_ENTRY_INTR_INFO, 0); + + if ( v->arch.hvm.vmx.exec_control & CPU_BASED_ACTIVATE_MSR_BITMAP ) + __vmwrite(MSR_BITMAP, virt_to_maddr(v->arch.hvm.vmx.msr_bitmap)); } static void sync_exception_state(struct vcpu *v) diff --git a/xen/include/asm-x86/hvm/vmx/vvmx.h b/xen/include/asm-x86/hvm/vmx/vvmx.h index 6b9c4ae0b2..c41f089939 100644 --- a/xen/include/asm-x86/hvm/vmx/vvmx.h +++ b/xen/include/asm-x86/hvm/vmx/vvmx.h @@ -37,7 +37,8 @@ struct nestedvmx { */ paddr_t vmxon_region_pa; void *iobitmap[2]; /* map (va) of L1 guest I/O bitmap */ - void *msrbitmap; /* map (va) of L1 guest MSR bitmap */ + struct vmx_msr_bitmap *msrbitmap; /* map (va) of L1 guest MSR bitmap */ + struct page_info *msr_merged; /* merged L1 and L2 MSR bitmap */ /* deferred nested interrupt */ struct { unsigned long intr_info; From patchwork Mon Feb 3 17:37:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 11363133 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2A17B138D for ; Mon, 3 Feb 2020 17:38:48 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 07BC42080C for ; Mon, 3 Feb 2020 17:38:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="H3O4Qmf/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 07BC42080C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iyffX-0007Se-2u; Mon, 03 Feb 2020 17:37:55 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iyffW-0007SK-8A for xen-devel@lists.xenproject.org; Mon, 03 Feb 2020 17:37:54 +0000 X-Inumbo-ID: e7467682-46ab-11ea-8e7c-12813bfff9fa Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id e7467682-46ab-11ea-8e7c-12813bfff9fa; Mon, 03 Feb 2020 17:37:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1580751473; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OqfmrT1+SOmdNfWCuLumoSoTJGFTwZPLnOQx0BuYkBQ=; b=H3O4Qmf/nR4iZYS/rUlMZyKj+O4nxkLIpnQxZlLzGvd09T6QiQs9/4SE CHtzEgb+5wsgN6GVz1m81D/OmYb8f+qO+Ck6wWQd1dddjzOx6gIYF/9t3 6wTYzIINaZuO2T1i0x24hIrj1jgtnYfl0JoKssmGwspRjJgrsFzCRIBV6 I=; Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=roger.pau@citrix.com; spf=Pass smtp.mailfrom=roger.pau@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa1.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa1.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa1.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa1.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa1.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa1.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: KWXFiGCuGdZxbr3kANVpz4QVt1xVV6ZRpG++9CFC+93pLOg6O3+inC8CJAv7TZKszQv4uxYNxf qKxVQ63zRzq9uRBFKsf/c1dYNMtfUJZghL+g1i4XfWQF4BboIvwE+u5+XYr9oDQbBhZvHqaQfH 1PlpFgBvcdoQsrFURuXi8ICiAAQAtcqjQN0d4JppNrK2v9YRYPp8sZUxGHkcr4O8jhsZYWZtPs fBuoywbOcEBdwhrZnI/99el/4+zcDeq515iPajVl6vR9+5ZwKRJW+VtmIdapdv7bp6TynK6TqA RAE= X-SBRS: 2.7 X-MesageID: 12036630 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.70,398,1574139600"; d="scan'208";a="12036630" From: Roger Pau Monne To: Date: Mon, 3 Feb 2020 18:37:28 +0100 Message-ID: <20200203173728.18135-3-roger.pau@citrix.com> X-Mailer: git-send-email 2.25.0 In-Reply-To: <20200203173728.18135-1-roger.pau@citrix.com> References: <20200203173728.18135-1-roger.pau@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 2/2] nvmx: always trap accesses to x2APIC MSRs X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Kevin Tian , Jun Nakajima , Wei Liu , Andrew Cooper , Roger Pau Monne Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Nested VMX doesn't expose support for SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE, SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY or SECONDARY_EXEC_APIC_REGISTER_VIRT, and hence the x2APIC MSRs should always be trapped in the nested guest MSR bitmap, or else a nested guest could access the hardware x2APIC MSRs given certain conditions. Accessing the hardware MSRs could be achieved by forcing the L0 Xen to use SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE and SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY or SECONDARY_EXEC_APIC_REGISTER_VIRT (if supported), and then creating a L2 guest with a MSR bitmap that doesn't trap accesses to the x2APIC MSR range. Then OR'ing both L0 and L1 MSR bitmaps would result in a bitmap that doesn't trap certain x2APIC MSRs and a VMCS that doesn't have SECONDARY_EXEC_VIRTUALIZE_X2APIC_MODE and SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY or SECONDARY_EXEC_APIC_REGISTER_VIRT set either. Fix this by making sure x2APIC MSRs are always trapped in the nested MSR bitmap. Signed-off-by: Roger Pau Monné Reviewed-by: Kevin Tian --- Changes since v1: - New in this version (split from #1 patch). - Use non-locked set_bit. --- xen/arch/x86/hvm/vmx/vvmx.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c index f118f88683..89ba2a80d5 100644 --- a/xen/arch/x86/hvm/vmx/vvmx.c +++ b/xen/arch/x86/hvm/vmx/vvmx.c @@ -567,6 +567,7 @@ static void update_msrbitmap(struct vcpu *v, uint32_t shadow_ctrl) { struct nestedvmx *nvmx = &vcpu_2_nvmx(v); struct vmx_msr_bitmap *msr_bitmap; + unsigned int msr; if ( !(shadow_ctrl & CPU_BASED_ACTIVATE_MSR_BITMAP) || !nvmx->msrbitmap ) @@ -587,6 +588,16 @@ static void update_msrbitmap(struct vcpu *v, uint32_t shadow_ctrl) v->arch.hvm.vmx.msr_bitmap->write_high, sizeof(msr_bitmap->write_high) * 8); + /* + * Nested VMX doesn't support any x2APIC hardware virtualization, so + * make sure all the x2APIC MSRs are trapped. + */ + for ( msr = MSR_X2APIC_FIRST; msr <= MSR_X2APIC_FIRST + 0xff; msr++ ) + { + __set_bit(msr, msr_bitmap->read_low); + __set_bit(msr, msr_bitmap->write_low); + } + unmap_domain_page(msr_bitmap); __vmwrite(MSR_BITMAP, page_to_maddr(nvmx->msr_merged));