From patchwork Wed Sep 25 02:41:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 11159929 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 955F41668 for ; Wed, 25 Sep 2019 02:43:17 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6FC3A2064A for ; Wed, 25 Sep 2019 02:43:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="iHRIbzba" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6FC3A2064A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iCxFJ-0000sm-QL; Wed, 25 Sep 2019 02:41:37 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iCxFI-0000sc-5M for xen-devel@lists.xenproject.org; Wed, 25 Sep 2019 02:41:36 +0000 X-Inumbo-ID: fdc62ae0-df3d-11e9-bf31-bc764e2007e4 Received: from out2-smtp.messagingengine.com (unknown [66.111.4.26]) by localhost (Halon) with ESMTPS id fdc62ae0-df3d-11e9-bf31-bc764e2007e4; Wed, 25 Sep 2019 02:41:35 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 2FC252222B; Tue, 24 Sep 2019 22:41:35 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Tue, 24 Sep 2019 22:41:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=jYpAjnlE+xFbalsg2q3WrQQg+q42vr+5BpmVq4v5g 5Q=; b=iHRIbzbaXTs40euYMt/wQSEQN4QJW6ZNvRP7c9Q3s+IDlSbB7fHq7gsWL MAExL9AenZmX1JB5dx1w5j5jxX3PJh/mJlBCtAEhE49EgPDKR7QwogrzXJEyKFgt hOjH9/hhoDvHXnoO1dpeHAxm3KNgj1JXwBq2prbiqv/kLZmy6D+Rk+mGLKyaFAPA Kc1tKTKtqHVafN66slkk8Hr0DwxcceCC+xZCjUNYj4ZcCXmkvMzFxVT9i4HqD5oY nV+bg2r/j6AFhLoFJU8GWeRrw7Dx7Wgt+UFiqhESP4sNWJzUpqpYeTNvK0At5YT2 2yZotfYfJgDkDwmdXxi7tmudBexsA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedufedrfedugdeivdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucfkphepledurdeihedrfeegrdef feenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghksehinhhvihhsihgslh gvthhhihhnghhslhgrsgdrtghomhenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from localhost.localdomain (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id AA369D6005E; Tue, 24 Sep 2019 22:41:33 -0400 (EDT) From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Date: Wed, 25 Sep 2019 04:41:23 +0200 Message-Id: <0d3f56a22a7f01024d0d83ed83028b37648f6a30.1569379186.git-series.marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v7 1/4] libxl: do not attach xen-pciback to HVM domain, if stubdomain is in use X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Anthony PERARD , Ian Jackson , Wei Liu , =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Wei Liu Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" HVM domains use IOMMU and device model assistance for communicating with PCI devices, xen-pcifront/pciback isn't directly needed by HVM domain. But pciback serve also second function - it reset the device when it is deassigned from the guest and for this reason pciback needs to be used with HVM domain too. When HVM domain has device model in stubdomain, attaching xen-pciback to the target domain itself may prevent attaching xen-pciback to the (PV) stubdomain, effectively breaking PCI passthrough. Fix this by attaching pciback only to one domain: if PV stubdomain is in use, let it be stubdomain (the commit prevents attaching device to target HVM in this case); otherwise, attach it to the target domain. Signed-off-by: Marek Marczykowski-Górecki Acked-by: Wei Liu --- Changes in v2: - previously called "libxl: attach xen-pciback only to PV domains" - instead of excluding all HVMs, change the condition to what actually matters here - check if stubdomain is in use; this way xen-pciback is always in use (either for the target domain, or it's stubdomain), fixing PCI reset by xen-pciback concerns Changes in v3: - adjust commit message --- tools/libxl/libxl_pci.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 03beb86..2e06a45 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -1106,7 +1106,7 @@ out: } } - if (!starting) + if (!starting && !libxl_get_stubdom_id(CTX, domid)) rc = libxl__device_pci_add_xenstore(gc, domid, pcidev, starting); else rc = 0; @@ -1302,7 +1302,7 @@ static void libxl__add_pcidevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid, } } - if (d_config->num_pcidevs > 0) { + if (d_config->num_pcidevs > 0 && !libxl_get_stubdom_id(CTX, domid)) { rc = libxl__create_pci_backend(gc, domid, d_config->pcidevs, d_config->num_pcidevs); if (rc < 0) { From patchwork Wed Sep 25 02:41:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 11159933 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3E1CB1800 for ; Wed, 25 Sep 2019 02:43:18 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 19CE52064A for ; Wed, 25 Sep 2019 02:43:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="TP2ATDiw" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 19CE52064A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iCxFL-0000ss-45; Wed, 25 Sep 2019 02:41:39 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iCxFJ-0000sh-HV for xen-devel@lists.xenproject.org; Wed, 25 Sep 2019 02:41:37 +0000 X-Inumbo-ID: fdcedf01-df3d-11e9-9627-12813bfff9fa Received: from out2-smtp.messagingengine.com (unknown [66.111.4.26]) by localhost (Halon) with ESMTPS id fdcedf01-df3d-11e9-9627-12813bfff9fa; Wed, 25 Sep 2019 02:41:35 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id B680522237; Tue, 24 Sep 2019 22:41:35 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Tue, 24 Sep 2019 22:41:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=1A8lEmAUpw019VXdWnEGZkLXEHyBw42z6uUjfYtGZ q4=; b=TP2ATDiwzgpnROPXuxqik48FXQS0X726acH61nBEcnSIJ/BDJRplkYddu poiLK7bIxbdbdvzzx6ufNSwRSBeg2geuvnCK8H/cYsC70q7P7KU8MHIm7byHZNcu U3Y4h74S1NBlzyQhcYo6+nZBTLSvdzVh1LY2PDFDYdNqOoyalYM/GPapKvGygbjy Zc/q5TPJASChx04PJlW1vKUvBvv8e7oJEU9uBCs8+WB7Ufe+Vf4Po0w0DPeazpe/ tpvCjGvlwsMevh2zYG0SRNuDM6q9ZIh0s5Bc0yClKlBPKA90Qyi6yhmS/SOIO2+D iSIjWIRWXHW8/5cVuc5zCZ3rdnHgA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedufedrfedugdeivdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucffohhmrghinhepvhgvrhhsihho nhdrqddqqdhtohholhhsnecukfhppeeluddrieehrdefgedrfeefnecurfgrrhgrmhepmh grihhlfhhrohhmpehmrghrmhgrrhgvkhesihhnvhhishhisghlvghthhhinhhgshhlrggs rdgtohhmnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from localhost.localdomain (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id A8BB9D6005B; Tue, 24 Sep 2019 22:41:34 -0400 (EDT) From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Date: Wed, 25 Sep 2019 04:41:24 +0200 Message-Id: X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v7 2/4] libxl: attach PCI device to qemu only after setting pciback/pcifront X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Anthony PERARD , Ian Jackson , Wei Liu , =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Wei Liu Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" When qemu is running in stubdomain, handling "pci-ins" command will fail if pcifront is not initialized already. Fix this by sending such command only after confirming that pciback/front is running. Signed-off-by: Marek Marczykowski-Górecki Acked-by: Wei Liu --- Changes in v2: - Fixed code style since previous version. --- tools/libxl/libxl_pci.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 2e06a45..578535f 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -1191,6 +1191,7 @@ int libxl__device_pci_add(libxl__gc *gc, uint32_t domid, libxl_device_pci *pcide { libxl_ctx *ctx = libxl__gc_owner(gc); unsigned int orig_vdev, pfunc_mask; + char *be_path; libxl_device_pci *assigned; int num_assigned, i, rc; int stubdomid = 0; @@ -1245,6 +1246,14 @@ int libxl__device_pci_add(libxl__gc *gc, uint32_t domid, libxl_device_pci *pcide rc = do_pci_add(gc, stubdomid, &pcidev_s, 0); if ( rc ) goto out; + /* Wait for the device actually being connected, otherwise device model + * running there will fail to find the device. */ + be_path = libxl__sprintf(gc, "%s/backend/pci/%d/0", + libxl__xs_get_dompath(gc, 0), stubdomid); + rc = libxl__wait_for_backend(gc, be_path, + GCSPRINTF("%d", XenbusStateConnected)); + if (rc) + goto out; } orig_vdev = pcidev->vdevfn & ~7U; From patchwork Wed Sep 25 02:41:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 11159935 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5A88217EE for ; Wed, 25 Sep 2019 02:43:18 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3666F2146E for ; Wed, 25 Sep 2019 02:43:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="jBnWMIbV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3666F2146E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iCxFO-0000tC-E3; Wed, 25 Sep 2019 02:41:42 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iCxFN-0000t3-5L for xen-devel@lists.xenproject.org; Wed, 25 Sep 2019 02:41:41 +0000 X-Inumbo-ID: fe8da3c2-df3d-11e9-bf31-bc764e2007e4 Received: from out2-smtp.messagingengine.com (unknown [66.111.4.26]) by localhost (Halon) with ESMTPS id fe8da3c2-df3d-11e9-bf31-bc764e2007e4; Wed, 25 Sep 2019 02:41:36 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id BE683222E4; Tue, 24 Sep 2019 22:41:36 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Tue, 24 Sep 2019 22:41:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=pjHm9ceYnWR9V0AqmOopig5VJRNLK4kBvvWCoPBaS OE=; b=jBnWMIbVeD7N7kfdGlLixZ9VDGAAMPpHAPOY78UDrmJKS67GNV+xbdUhB Sfo/CRoNvvfCVzGVx/ij6XhW0kyMzbcCyWCAuAqzs1FukPHYOJWla9jw6yHERSxC Z6Lw+mShONi2fc7D/W5P0pe64vKXQMeuURkA8Y5zGjGqw1VMGLMR99PkUZjJD1t8 FLNU9RFjg9jyrbN6cwYbGGyBWetNK7JRfN8WHUCQbPNl6niFWyMhFGu4o1EeB0s+ hX66XYvKbJdRgEBxpTCiX6pz88+yNE+FEwEl0Qa5eC+Wnr9PNWcVcYoQtAFINvZE oLB0FSgfe09ubQKoFhf905B4FVh/Q== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedufedrfedugdeivdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucfkphepledurdeihedrfeegrdef feenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghksehinhhvihhsihgslh gvthhhihhnghhslhgrsgdrtghomhenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from localhost.localdomain (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id AB2AED6005A; Tue, 24 Sep 2019 22:41:35 -0400 (EDT) From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Date: Wed, 25 Sep 2019 04:41:25 +0200 Message-Id: <206c62834cf039074d95c869eebdec333471bf1f.1569379186.git-series.marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v7 3/4] libxl: don't try to manipulate json config for stubdomain X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Anthony PERARD , Ian Jackson , Wei Liu , =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Wei Liu Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Stubdomain do not have it's own config file - its configuration is derived from target domains. Do not try to manipulate it when attaching PCI device. This bug prevented starting HVM with stubdomain and PCI passthrough device attached. Signed-off-by: Marek Marczykowski-Górecki Acked-by: Wei Liu --- Changes in v3: - skip libxl__dm_check_start too, as stubdomain is guaranteed to be running at this stage already - do not init d_config at all, as it is used only for json manipulation Changes in v4: - adjust comment style --- tools/libxl/libxl_pci.c | 50 ++++++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 578535f..d26fc9a 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -120,10 +120,14 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d libxl_domain_config d_config; libxl_device_pci pcidev_saved; libxl__domain_userdata_lock *lock = NULL; + bool is_stubdomain = libxl_is_stubdom(CTX, domid, NULL); - libxl_domain_config_init(&d_config); - libxl_device_pci_init(&pcidev_saved); - libxl_device_pci_copy(CTX, &pcidev_saved, pcidev); + /* Stubdomain doesn't have own config. */ + if (!is_stubdomain) { + libxl_domain_config_init(&d_config); + libxl_device_pci_init(&pcidev_saved); + libxl_device_pci_copy(CTX, &pcidev_saved, pcidev); + } be_path = libxl__domain_device_backend_path(gc, 0, domid, 0, LIBXL__DEVICE_KIND_PCI); @@ -152,27 +156,35 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d GCNEW(device); libxl__device_from_pcidev(gc, domid, pcidev, device); - lock = libxl__lock_domain_userdata(gc, domid); - if (!lock) { - rc = ERROR_LOCK_FAIL; - goto out; - } + /* + * Stubdomin config is derived from its target domain, it doesn't have + * its own file. + */ + if (!is_stubdomain) { + lock = libxl__lock_domain_userdata(gc, domid); + if (!lock) { + rc = ERROR_LOCK_FAIL; + goto out; + } - rc = libxl__get_domain_configuration(gc, domid, &d_config); - if (rc) goto out; + rc = libxl__get_domain_configuration(gc, domid, &d_config); + if (rc) goto out; - device_add_domain_config(gc, &d_config, &libxl__pcidev_devtype, - &pcidev_saved); + device_add_domain_config(gc, &d_config, &libxl__pcidev_devtype, + &pcidev_saved); - rc = libxl__dm_check_start(gc, &d_config, domid); - if (rc) goto out; + rc = libxl__dm_check_start(gc, &d_config, domid); + if (rc) goto out; + } for (;;) { rc = libxl__xs_transaction_start(gc, &t); if (rc) goto out; - rc = libxl__set_domain_configuration(gc, domid, &d_config); - if (rc) goto out; + if (lock) { + rc = libxl__set_domain_configuration(gc, domid, &d_config); + if (rc) goto out; + } libxl__xs_writev(gc, t, be_path, libxl__xs_kvs_of_flexarray(gc, back)); @@ -184,8 +196,10 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d out: libxl__xs_transaction_abort(gc, &t); if (lock) libxl__unlock_domain_userdata(lock); - libxl_device_pci_dispose(&pcidev_saved); - libxl_domain_config_dispose(&d_config); + if (!is_stubdomain) { + libxl_device_pci_dispose(&pcidev_saved); + libxl_domain_config_dispose(&d_config); + } return rc; } From patchwork Wed Sep 25 02:41:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 11159937 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 854C01599 for ; Wed, 25 Sep 2019 02:43:18 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 557522064A for ; Wed, 25 Sep 2019 02:43:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="NOOzlYJ/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 557522064A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iCxFV-0000vE-23; Wed, 25 Sep 2019 02:41:49 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iCxFT-0000ux-F7 for xen-devel@lists.xenproject.org; Wed, 25 Sep 2019 02:41:47 +0000 X-Inumbo-ID: fff168c0-df3d-11e9-9627-12813bfff9fa Received: from new1-smtp.messagingengine.com (unknown [66.111.4.221]) by localhost (Halon) with ESMTPS id fff168c0-df3d-11e9-9627-12813bfff9fa; Wed, 25 Sep 2019 02:41:39 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailnew.nyi.internal (Postfix) with ESMTP id D8DE61AC0; Tue, 24 Sep 2019 22:41:38 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Tue, 24 Sep 2019 22:41:38 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=56HX9QWMCTOordPBpOUNx2/N7BYQBZwgA4GsfLolr ew=; b=NOOzlYJ/Md86GE8/ahL/vzyt/hsjXijoCVYhhqehQssD1Mqs0kAuyeY/B QBylCkdo4YyegQzSgIu1fgWaKmTZEZ1NvQ1RSh/4k9Kf3OR20d8gv1aFYrNEeJfQ Gfm6GKcVczz5FHh/LF5PTNFPInxmuukGB3N3cfeQp0fNOK1isVj6/m8gWWf6EDQU GX9lpgM6IZQkDNPGtM6c2fSZAmg71wV0V27BGhOHccnEzi2jfcpBgu7OQWGKs6JY sOORCZDmlNB51iWri+vgXc0s/aj4JmWW+adqa+ccyj2OoUbEdHOdNf625UHumzL4 tC24+fzeNOlFAQK7Tg/7GmGqd651Q== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedufedrfedugdeivdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucffohhmrghinhepghhithhhuhgs rdgtohhmnecukfhppeeluddrieehrdefgedrfeefnecurfgrrhgrmhepmhgrihhlfhhroh hmpehmrghrmhgrrhgvkhesihhnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhmnecu vehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from localhost.localdomain (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id AF841D60062; Tue, 24 Sep 2019 22:41:36 -0400 (EDT) From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Date: Wed, 25 Sep 2019 04:41:26 +0200 Message-Id: <7d011094eed3f5c3cf6971cc8760874fd56ca443.1569379186.git-series.marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v7 4/4] xen/x86: Allow stubdom access to irq created for msi. X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Kevin Tian , Stefano Stabellini , Suravee Suthikulpanit , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , =?utf-8?q?Marek_Marczykowski-G?= =?utf-8?q?=C3=B3recki?= , Tim Deegan , Simon Gaiser , Julien Grall , Jan Beulich , Brian Woods , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Stubdomains need to be given sufficient privilege over the guest which it provides emulation for in order for PCI passthrough to work correctly. When a HVM domain try to enable MSI, QEMU in stubdomain calls PHYSDEVOP_map_pirq, but later it needs to call XEN_DOMCTL_bind_pt_irq as part of xc_domain_update_msi_irq. Allow for that as part of PHYSDEVOP_map_pirq. This is not needed for PCI INTx, because IRQ in that case is known beforehand and the stubdomain is given permissions over this IRQ by libxl__device_pci_add (there's a do_pci_add against the stubdomain). create_irq() already grant IRQ access to hardware_domain, with assumption the device model (something managing this IRQ) lives there. Modify create_irq() to take additional parameter pointing at device model domain - which may be dom0 or stubdomain. Save ID of the domain given permission, to revoke it in destroy_irq() - easier and cleaner than replaying logic of create_irq() parameter. Use domid instead of actual reference to the domain, because it might get destroyed before destroying IRQ (stubdomain is destroyed before its target domain). And it is not an issue, because IRQ permissions live within domain structure, so destroying a domain also implicitly revoke the permission. Potential domid reuse is detected by by checking if that domain does have permission over the IRQ being destroyed. Then, adjust all callers to provide the parameter. In case of calls not related to stubdomain-initiated allocations, give it either hardware_domain (so the behavior is unchanged there), or NULL for interrupts used by Xen internally. Inspired by https://github.com/OpenXT/xenclient-oe/blob/5e0e7304a5a3c75ef01240a1e3673665b2aaf05e/recipes-extended/xen/files/stubdomain-msi-irq-access.patch by Eric Chanudet . Signed-off-by: Simon Gaiser Signed-off-by: Marek Marczykowski-Górecki Reviewed-by: Roger Pau Monné --- Changes in v3: - extend commit message Changes in v4: - add missing destroy_irq on error path Changes in v5: - move irq_{grant,revoke}_access() to {create,destroy}_irq(), which basically make it a different patch - add get_dm_domain() helper - do not give hardware_domain permission over IRQs used in Xen internally - rename create_irq argument to just 'd', to avoid confusion when it's called by hardware domain - verify that device is de-assigned before pci_remove_device call - save ID of domain given permission in create_irq(), to revoke it in destroy_irq() - drop domain parameter from destroy_irq() and msi_free_irq() - do not give hardware domain permission over IRQ created in iommu_set_interrupt() Changes in v6: - do not give permission over hpet irq to hardware_domain - move creator_domid to arch_irq_desc - fix creator_domid initialization - always give current->domain permission instead of using get_dm_domain() helper. Analysis of all its use cases tells that it is the only value it returns. - drop unrelated change Changes in v7: - Code style improvements (spaces, use %pd etc) - use bool parameter to create_irq, as it's only getting current->domain or NULL - remove redundant irq_access_permitted() --- xen/arch/x86/hpet.c | 3 +- xen/arch/x86/irq.c | 42 +++++++++++++++++-------- xen/drivers/char/ns16550.c | 2 +- xen/drivers/passthrough/amd/iommu_init.c | 2 +- xen/drivers/passthrough/vtd/iommu.c | 3 +- xen/include/asm-x86/irq.h | 7 +++- 6 files changed, 41 insertions(+), 18 deletions(-) diff --git a/xen/arch/x86/hpet.c b/xen/arch/x86/hpet.c index 4b08488..57f68fa 100644 --- a/xen/arch/x86/hpet.c +++ b/xen/arch/x86/hpet.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -368,7 +369,7 @@ static int __init hpet_assign_irq(struct hpet_event_channel *ch) { int irq; - if ( (irq = create_irq(NUMA_NO_NODE)) < 0 ) + if ( (irq = create_irq(NUMA_NO_NODE, false)) < 0 ) return irq; ch->msi.irq = irq; diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c index 0ee3346..256dd02 100644 --- a/xen/arch/x86/irq.c +++ b/xen/arch/x86/irq.c @@ -254,7 +254,8 @@ void __init clear_irq_vector(int irq) /* * Dynamic irq allocate and deallocation for MSI */ -int create_irq(nodeid_t node) + +int create_irq(nodeid_t node, bool grant_access) { int irq, ret; struct irq_desc *desc; @@ -282,18 +283,23 @@ int create_irq(nodeid_t node) } ret = assign_irq_vector(irq, mask); } + + ASSERT(desc->arch.creator_domid == DOMID_INVALID); + if (ret < 0) { desc->arch.used = IRQ_UNUSED; irq = ret; } - else if ( hardware_domain ) + else if ( grant_access ) { - ret = irq_permit_access(hardware_domain, irq); + ret = irq_permit_access(current->domain, irq); if ( ret ) printk(XENLOG_G_ERR - "Could not grant Dom0 access to IRQ%d (error %d)\n", - irq, ret); + "Could not grant %pd access to IRQ%d (error %d)\n", + current->domain, irq, ret); + else + desc->arch.creator_domid = current->domain->domain_id; } return irq; @@ -307,14 +313,23 @@ void destroy_irq(unsigned int irq) BUG_ON(!MSI_IRQ(irq)); - if ( hardware_domain ) + if ( desc->arch.creator_domid != DOMID_INVALID ) { - int err = irq_deny_access(hardware_domain, irq); + struct domain *d = get_domain_by_id(desc->arch.creator_domid); - if ( err ) - printk(XENLOG_G_ERR - "Could not revoke Dom0 access to IRQ%u (error %d)\n", - irq, err); + if ( d ) + { + int err = irq_deny_access(d, irq); + if ( err ) + printk(XENLOG_G_ERR + "Could not revoke %pd access to IRQ%u (error %d)\n", + d, irq, err); + } + + if ( d ) + put_domain(d); + + desc->arch.creator_domid = DOMID_INVALID; } spin_lock_irqsave(&desc->lock, flags); @@ -381,6 +396,7 @@ int arch_init_one_irq_desc(struct irq_desc *desc) desc->arch.vector = IRQ_VECTOR_UNASSIGNED; desc->arch.old_vector = IRQ_VECTOR_UNASSIGNED; + desc->arch.creator_domid = DOMID_INVALID; return 0; } @@ -2133,7 +2149,7 @@ int map_domain_pirq( spin_unlock_irqrestore(&desc->lock, flags); info = NULL; - irq = create_irq(NUMA_NO_NODE); + irq = create_irq(NUMA_NO_NODE, true); ret = irq >= 0 ? prepare_domain_irq_pirq(d, irq, pirq + nr, &info) : irq; if ( ret < 0 ) @@ -2818,7 +2834,7 @@ int allocate_and_map_msi_pirq(struct domain *d, int index, int *pirq_p, if ( irq == -1 ) { case MAP_PIRQ_TYPE_MULTI_MSI: - irq = create_irq(NUMA_NO_NODE); + irq = create_irq(NUMA_NO_NODE, true); } if ( irq < nr_irqs_gsi || irq >= nr_irqs ) diff --git a/xen/drivers/char/ns16550.c b/xen/drivers/char/ns16550.c index 8667de6..fcd3979 100644 --- a/xen/drivers/char/ns16550.c +++ b/xen/drivers/char/ns16550.c @@ -722,7 +722,7 @@ static void __init ns16550_init_irq(struct serial_port *port) struct ns16550 *uart = port->uart; if ( uart->msi ) - uart->irq = create_irq(0); + uart->irq = create_irq(0, false); #endif } diff --git a/xen/drivers/passthrough/amd/iommu_init.c b/xen/drivers/passthrough/amd/iommu_init.c index bb9f33e..233a8ae 100644 --- a/xen/drivers/passthrough/amd/iommu_init.c +++ b/xen/drivers/passthrough/amd/iommu_init.c @@ -765,7 +765,7 @@ static bool_t __init set_iommu_interrupt_handler(struct amd_iommu *iommu) { int irq, ret; - irq = create_irq(NUMA_NO_NODE); + irq = create_irq(NUMA_NO_NODE, false); if ( irq <= 0 ) { dprintk(XENLOG_ERR, "IOMMU: no irqs\n"); diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c index 5d72270..24a1e92 100644 --- a/xen/drivers/passthrough/vtd/iommu.c +++ b/xen/drivers/passthrough/vtd/iommu.c @@ -1138,7 +1138,8 @@ static int __init iommu_set_interrupt(struct acpi_drhd_unit *drhd) struct irq_desc *desc; irq = create_irq(rhsa ? pxm_to_node(rhsa->proximity_domain) - : NUMA_NO_NODE); + : NUMA_NO_NODE, + false); if ( irq <= 0 ) { dprintk(XENLOG_ERR VTDPREFIX, "IOMMU: no irq available!\n"); diff --git a/xen/include/asm-x86/irq.h b/xen/include/asm-x86/irq.h index bc0c0c1..79853d0 100644 --- a/xen/include/asm-x86/irq.h +++ b/xen/include/asm-x86/irq.h @@ -45,6 +45,11 @@ struct arch_irq_desc { unsigned move_cleanup_count; u8 move_in_progress : 1; s8 used; + /* + * Weak reference to domain having permission over this IRQ (which can + * be different from the domain actually havint the IRQ assigned) + */ + domid_t creator_domid; }; /* For use with irq_desc.arch.used */ @@ -161,7 +166,7 @@ int init_irq_data(void); void clear_irq_vector(int irq); int irq_to_vector(int irq); -int create_irq(nodeid_t node); +int create_irq(nodeid_t node, bool grant_access); void destroy_irq(unsigned int irq); int assign_irq_vector(int irq, const cpumask_t *);