From patchwork Wed Mar 11 18:34:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrew Cooper X-Patchwork-Id: 11432517 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 98958921 for ; Wed, 11 Mar 2020 18:36:14 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 29BA1206E9 for ; Wed, 11 Mar 2020 18:36:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="NznS6QNE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 29BA1206E9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1jC6C9-00023m-4W; Wed, 11 Mar 2020 18:35:05 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1jC6C7-00023h-QW for xen-devel@lists.xenproject.org; Wed, 11 Mar 2020 18:35:03 +0000 X-Inumbo-ID: 035a2cab-63c7-11ea-b048-12813bfff9fa Received: from esa6.hc3370-68.iphmx.com (unknown [216.71.155.175]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 035a2cab-63c7-11ea-b048-12813bfff9fa; Wed, 11 Mar 2020 18:35:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1583951700; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=vN3MrrVLsdG0ZXZ0Rb8ZSmg8dcJzfgGu3sWyXZg2pG8=; b=NznS6QNE0PLET+R7MFJhOZcT++R4l729NECa3V4H7zoV1B4axLSCRWcm ET2NHztSy3H+AIPrTadwobfBdXwL/PXeiz4VyJ3Hg+d6T0H9XZvcgQYV8 QUj0SLaqpcof3OlpC2IVcZlw34SSxYlxCP0B2NYgzcPrTgdtfXOgmfl5I A=; Authentication-Results: esa6.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=andrew.cooper3@citrix.com; spf=Pass smtp.mailfrom=Andrew.Cooper3@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa6.hc3370-68.iphmx.com: no sender authenticity information available from domain of andrew.cooper3@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa6.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="andrew.cooper3@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa6.hc3370-68.iphmx.com: domain of Andrew.Cooper3@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa6.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="Andrew.Cooper3@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa6.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa6.hc3370-68.iphmx.com; envelope-from="Andrew.Cooper3@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: IyzGp3cfeIZFUNXD+cf2kmPk/xs0P+vUSoU+S09aNcidd3R4Y/4AKOQa1dUeOXZtCVSPJ7K+2b r4vBP1GG394rhZjmy9eS4ero6fkA009ohWYlspM25UKSFNaOLXwSYEClnh99PuSbNyRsVkuJHp kFpC5BmqQGQvjmtU9JECBu2OveoS6pc8gOpBsL1qRvJBZFclmwMJ4Xiy0Cw3VqlAblp7g9BfIF veRRmB+PaK7QRAPgm7r/T3KtOT7myYYOgpd3/wHs94NsSluGyx9hb8CGnsGJiHddKBabF6ktG3 QsE= X-SBRS: 2.7 X-MesageID: 14211912 X-Ironport-Server: esa6.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.70,541,1574139600"; d="scan'208";a="14211912" From: Andrew Cooper To: Xen-devel Date: Wed, 11 Mar 2020 18:34:55 +0000 Message-ID: <20200311183455.23729-1-andrew.cooper3@citrix.com> X-Mailer: git-send-email 2.11.0 MIME-Version: 1.0 Subject: [Xen-devel] [PATCH] x86/vvmx: Fix deadlock with MSR bitmap merging X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Andrew Cooper , Kevin Tian , Wei Liu , Jan Beulich , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" c/s c47984aabead "nvmx: implement support for MSR bitmaps" introduced a use of map_domain_page() which may get used in the middle of context switch. This is not safe, and causes Xen to deadlock on the mapcache lock: (XEN) Xen call trace: (XEN) [] R _spin_lock+0x34/0x5e (XEN) [] F map_domain_page+0x250/0x527 (XEN) [] F do_page_fault+0x420/0x780 (XEN) [] F x86_64/entry.S#handle_exception_saved+0x68/0x94 (XEN) [] F __find_next_zero_bit+0x28/0x69 (XEN) [] F map_domain_page+0x2c6/0x527 (XEN) [] F nvmx_update_exec_control+0x1d7/0x323 (XEN) [] F vmx_update_cpu_exec_control+0x23/0x40 (XEN) [] F arch/x86/hvm/vmx/vmx.c#vmx_ctxt_switch_from+0xb7/0x121 (XEN) [] F arch/x86/domain.c#__context_switch+0x124/0x4a9 (XEN) [] F context_switch+0x154/0x62c (XEN) [] F common/sched/core.c#sched_context_switch+0x16a/0x175 (XEN) [] F common/sched/core.c#schedule+0x2ad/0x2bc (XEN) [] F common/softirq.c#__do_softirq+0xb7/0xc8 (XEN) [] F do_softirq+0x18/0x1a (XEN) [] F vmx_asm_do_vmentry+0x2b/0x30 Convert the domheap page into being a xenheap page. Signed-off-by: Andrew Cooper Reviewed-by: Jan Beulich Reviewed-by: Kevin Tian --- CC: Jan Beulich CC: Wei Liu CC: Roger Pau Monné CC: Kevin Tian I suspect this is the not-quite-consistent-enough-to-bisect issue which OSSTest is hitting and interfering with pushes to master. --- xen/arch/x86/hvm/vmx/vvmx.c | 19 ++++--------------- xen/include/asm-x86/hvm/vmx/vvmx.h | 2 +- 2 files changed, 5 insertions(+), 16 deletions(-) diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c index 926a11c15f..f049920196 100644 --- a/xen/arch/x86/hvm/vmx/vvmx.c +++ b/xen/arch/x86/hvm/vmx/vvmx.c @@ -130,12 +130,9 @@ int nvmx_vcpu_initialise(struct vcpu *v) if ( cpu_has_vmx_msr_bitmap ) { - nvmx->msr_merged = alloc_domheap_page(d, MEMF_no_owner); + nvmx->msr_merged = alloc_xenheap_page(); if ( !nvmx->msr_merged ) - { - gdprintk(XENLOG_ERR, "nest: allocation for MSR bitmap failed\n"); return -ENOMEM; - } } nvmx->ept.enabled = 0; @@ -198,11 +195,7 @@ static void vcpu_relinquish_resources(struct vcpu *v) { struct nestedvmx *nvmx = &vcpu_2_nvmx(v); - if ( nvmx->msr_merged ) - { - free_domheap_page(nvmx->msr_merged); - nvmx->msr_merged = NULL; - } + FREE_XENHEAP_PAGE(nvmx->msr_merged); } void nvmx_domain_relinquish_resources(struct domain *d) @@ -575,14 +568,12 @@ unsigned long *_shadow_io_bitmap(struct vcpu *v) static void update_msrbitmap(struct vcpu *v, uint32_t shadow_ctrl) { struct nestedvmx *nvmx = &vcpu_2_nvmx(v); - struct vmx_msr_bitmap *msr_bitmap; + struct vmx_msr_bitmap *msr_bitmap = nvmx->msr_merged; if ( !(shadow_ctrl & CPU_BASED_ACTIVATE_MSR_BITMAP) || !nvmx->msrbitmap ) return; - msr_bitmap = __map_domain_page(nvmx->msr_merged); - bitmap_or(msr_bitmap->read_low, nvmx->msrbitmap->read_low, v->arch.hvm.vmx.msr_bitmap->read_low, sizeof(msr_bitmap->read_low) * 8); @@ -603,9 +594,7 @@ static void update_msrbitmap(struct vcpu *v, uint32_t shadow_ctrl) bitmap_set(msr_bitmap->read_low, MSR_X2APIC_FIRST, 0x100); bitmap_set(msr_bitmap->write_low, MSR_X2APIC_FIRST, 0x100); - unmap_domain_page(msr_bitmap); - - __vmwrite(MSR_BITMAP, page_to_maddr(nvmx->msr_merged)); + __vmwrite(MSR_BITMAP, virt_to_maddr(nvmx->msr_merged)); } void nvmx_update_exec_control(struct vcpu *v, u32 host_cntrl) diff --git a/xen/include/asm-x86/hvm/vmx/vvmx.h b/xen/include/asm-x86/hvm/vmx/vvmx.h index c41f089939..d5f68f30b1 100644 --- a/xen/include/asm-x86/hvm/vmx/vvmx.h +++ b/xen/include/asm-x86/hvm/vmx/vvmx.h @@ -38,7 +38,7 @@ struct nestedvmx { paddr_t vmxon_region_pa; void *iobitmap[2]; /* map (va) of L1 guest I/O bitmap */ struct vmx_msr_bitmap *msrbitmap; /* map (va) of L1 guest MSR bitmap */ - struct page_info *msr_merged; /* merged L1 and L2 MSR bitmap */ + struct vmx_msr_bitmap *msr_merged; /* merged L1 and L2 MSR bitmap */ /* deferred nested interrupt */ struct { unsigned long intr_info;