From patchwork Wed Jul 17 01:00:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 11046853 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 681EA6C5 for ; Wed, 17 Jul 2019 01:03:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5871F2872A for ; Wed, 17 Jul 2019 01:03:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4BBB5284D2; Wed, 17 Jul 2019 01:03:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 27600285ED for ; Wed, 17 Jul 2019 01:03:15 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYK1-00054L-Os; Wed, 17 Jul 2019 01:01:29 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYK0-000541-8k for xen-devel@lists.xenproject.org; Wed, 17 Jul 2019 01:01:28 +0000 X-Inumbo-ID: 65d730e4-a82e-11e9-a195-ef06a8b04646 Received: from wout4-smtp.messagingengine.com (unknown [64.147.123.20]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 65d730e4-a82e-11e9-a195-ef06a8b04646; Wed, 17 Jul 2019 01:01:24 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 4A3863F2; Tue, 16 Jul 2019 21:01:23 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Tue, 16 Jul 2019 21:01:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=T433qKPJ5yOL+6nQ2s/YupvBv1Cpvvjysd0Aung09 q0=; b=oyCGe7RDRE1pj7tvCAihJaqKTHefo2z0WgJzo3vHuI3OXe4zuP1OUJvNF 7EAs6vuqyxcqEhCDsLdudAssbQ0PKYOMYnxJ4rsWJWhC9rrxo7IWO+AOUxCD+ATu EG8NruvdIatlz+KsU+3P6jju4cy20qwG5AEHY3nmFXAkWOXBs8H6Iun+6HpWMs+F v52pMmeIUlLZpsQodiCK9s96jBZpaicQpMx/LDwG7Ld91oVUkcAnwZhOiEj3i5EV Thdd6sT6oNT1Xk/GoJtKdzp/9JCDgQB7F/XA3LtRKsrcrd+27gVC5PdqNb50sKY2 t/QTDOyE4GXRPUfB+w/rGHk6X/xqQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddriedugdegvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucfkphepledurdeihedrfeegrdef feenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghksehinhhvihhsihgslh gvthhhihhnghhslhgrsgdrtghomhenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from localhost.localdomain (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id 56EC9380084; Tue, 16 Jul 2019 21:01:21 -0400 (EDT) From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Date: Wed, 17 Jul 2019 03:00:39 +0200 Message-Id: <5b6a20c0b33a2b52175174a27ce388208f932992.1563325215.git-series.marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v5 1/6] libxl: do not attach xen-pciback to HVM domain, if stubdomain is in use X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Anthony PERARD , Ian Jackson , Wei Liu , =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Wei Liu Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP HVM domains use IOMMU and device model assistance for communicating with PCI devices, xen-pcifront/pciback isn't directly needed by HVM domain. But pciback serve also second function - it reset the device when it is deassigned from the guest and for this reason pciback needs to be used with HVM domain too. When HVM domain has device model in stubdomain, attaching xen-pciback to the target domain itself may prevent attaching xen-pciback to the (PV) stubdomain, effectively breaking PCI passthrough. Fix this by attaching pciback only to one domain: if PV stubdomain is in use, let it be stubdomain (the commit prevents attaching device to target HVM in this case); otherwise, attach it to the target domain. Signed-off-by: Marek Marczykowski-Górecki Acked-by: Wei Liu --- Changes in v2: - previously called "libxl: attach xen-pciback only to PV domains" - instead of excluding all HVMs, change the condition to what actually matters here - check if stubdomain is in use; this way xen-pciback is always in use (either for the target domain, or it's stubdomain), fixing PCI reset by xen-pciback concerns Changes in v3: - adjust commit message --- tools/libxl/libxl_pci.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 4ec6872..7ffab89 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -1106,7 +1106,7 @@ out: } } - if (!starting) + if (!starting && !libxl_get_stubdom_id(CTX, domid)) rc = libxl__device_pci_add_xenstore(gc, domid, pcidev, starting); else rc = 0; @@ -1302,7 +1302,7 @@ static void libxl__add_pcidevs(libxl__egc *egc, libxl__ao *ao, uint32_t domid, } } - if (d_config->num_pcidevs > 0) { + if (d_config->num_pcidevs > 0 && !libxl_get_stubdom_id(CTX, domid)) { rc = libxl__create_pci_backend(gc, domid, d_config->pcidevs, d_config->num_pcidevs); if (rc < 0) { From patchwork Wed Jul 17 01:00:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 11046865 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E28456C5 for ; Wed, 17 Jul 2019 01:06:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D09C628606 for ; Wed, 17 Jul 2019 01:06:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C1474286B9; Wed, 17 Jul 2019 01:06:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 61E8328606 for ; Wed, 17 Jul 2019 01:06:28 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYNG-0005eu-Sj; Wed, 17 Jul 2019 01:04:50 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYNE-0005el-R2 for xen-devel@lists.xenproject.org; Wed, 17 Jul 2019 01:04:48 +0000 X-Inumbo-ID: 6607eb62-a82e-11e9-ac5a-cb9da3a1b495 Received: from wout4-smtp.messagingengine.com (unknown [64.147.123.20]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 6607eb62-a82e-11e9-ac5a-cb9da3a1b495; Wed, 17 Jul 2019 01:01:24 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id CA2003FA; Tue, 16 Jul 2019 21:01:23 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Tue, 16 Jul 2019 21:01:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=4TEx41rBDtnAVuYynPevXwnuw1J6hoytIjvxfT36S /8=; b=pQ0Ka6Dqlol6W5Cy2p1L3KdFvBRIWRJKihyfJm3QSBfX8t1I26fL/Ucsc ou4rhRddxrXHwzDc8Am8P4c9AnJzdBadlB10YTbr8N7Ey7qZbacRDxO3eCpbAz6k o5Zt3iOeODGcLb1n+dWyYh7SHkommdCaf0e7QaS/gv9Stka0ZbWjlW6OpSHoLBKM Sn6FeBE/NviIt2odnbjEveHnHsa3oBPuZ/132mwaDNO0gYYgSJvaozF+jB3dgERS EomS7cKwh3XKn9nv2PTfyUApRKqihDYRz5UNhw5lxVj+FgLKtPXJxkVpsQvtOBnV 9fGZvS0eXQTl90w7FnhlZ1PEhAOig== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddriedugdegvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucffohhmrghinhepvhgvrhhsihho nhdrqddqqdhtohholhhsnecukfhppeeluddrieehrdefgedrfeefnecurfgrrhgrmhepmh grihhlfhhrohhmpehmrghrmhgrrhgvkhesihhnvhhishhisghlvghthhhinhhgshhlrggs rdgtohhmnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from localhost.localdomain (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id 64852380089; Tue, 16 Jul 2019 21:01:22 -0400 (EDT) From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Date: Wed, 17 Jul 2019 03:00:40 +0200 Message-Id: <9ced73c8ec8569b90cbdf413e5db2ada5121c973.1563325215.git-series.marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v5 2/6] libxl: attach PCI device to qemu only after setting pciback/pcifront X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Anthony PERARD , Ian Jackson , Wei Liu , =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Wei Liu Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP When qemu is running in stubdomain, handling "pci-ins" command will fail if pcifront is not initialized already. Fix this by sending such command only after confirming that pciback/front is running. Signed-off-by: Marek Marczykowski-Górecki Acked-by: Wei Liu --- Changes in v2: - Fixed code style since previous version. --- tools/libxl/libxl_pci.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 7ffab89..18089ea 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -1191,6 +1191,7 @@ int libxl__device_pci_add(libxl__gc *gc, uint32_t domid, libxl_device_pci *pcide { libxl_ctx *ctx = libxl__gc_owner(gc); unsigned int orig_vdev, pfunc_mask; + char *be_path; libxl_device_pci *assigned; int num_assigned, i, rc; int stubdomid = 0; @@ -1245,6 +1246,14 @@ int libxl__device_pci_add(libxl__gc *gc, uint32_t domid, libxl_device_pci *pcide rc = do_pci_add(gc, stubdomid, &pcidev_s, 0); if ( rc ) goto out; + /* Wait for the device actually being connected, otherwise device model + * running there will fail to find the device. */ + be_path = libxl__sprintf(gc, "%s/backend/pci/%d/0", + libxl__xs_get_dompath(gc, 0), stubdomid); + rc = libxl__wait_for_backend(gc, be_path, + GCSPRINTF("%d", XenbusStateConnected)); + if (rc) + goto out; } orig_vdev = pcidev->vdevfn & ~7U; From patchwork Wed Jul 17 01:00:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 11046855 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2EC44912 for ; Wed, 17 Jul 2019 01:03:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F590284D2 for ; Wed, 17 Jul 2019 01:03:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 132682871E; Wed, 17 Jul 2019 01:03:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 69356284D2 for ; Wed, 17 Jul 2019 01:03:16 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYK2-00054X-Ab; Wed, 17 Jul 2019 01:01:30 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYK0-000543-IR for xen-devel@lists.xenproject.org; Wed, 17 Jul 2019 01:01:28 +0000 X-Inumbo-ID: 66ac2038-a82e-11e9-b486-374560743dd7 Received: from wout4-smtp.messagingengine.com (unknown [64.147.123.20]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 66ac2038-a82e-11e9-b486-374560743dd7; Wed, 17 Jul 2019 01:01:26 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id E0987405; Tue, 16 Jul 2019 21:01:24 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Tue, 16 Jul 2019 21:01:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=EpWbvZmO3WWqGzuHKNIUKK+om3VcVBbW+Wmx/jZtn 28=; b=tfHi7rjG9i/5XmtKBagr9F7jklbSl3VMyiP81TDTr32w7dHTGCMEaRFV7 15N5OCVaREFjxgLD7dBSzPx5F5werNxIorFgoI7ZYB8NE3Lvj0hnd1jJGfSWWXlY Lm3JBykpAjDPLHN3Rz1UB9hFIxw6hxfgZ1vBKjOTv96INBZnHDnZwqfXnaRxe8Iw wJhKnKVOOnUyDwxM44xIqsyb4mJxKoTTL6zT+MkqJv98xLX3Li8lO32CL6psYu77 Ubd9V7TaDr0u3NvUAjBMGUdOpkaRLgp0R5hV60TyuXlRtnO17hHjVRkgXSiKzYHc 7J7aAYB0mbEis8YHyB7fQ3N6rL7dA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddriedugdegvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucfkphepledurdeihedrfeegrdef feenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghksehinhhvihhsihgslh gvthhhihhnghhslhgrsgdrtghomhenucevlhhushhtvghrufhiiigvpedu X-ME-Proxy: Received: from localhost.localdomain (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id 7041A38008A; Tue, 16 Jul 2019 21:01:23 -0400 (EDT) From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Date: Wed, 17 Jul 2019 03:00:41 +0200 Message-Id: X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v5 3/6] libxl: don't try to manipulate json config for stubdomain X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Anthony PERARD , Ian Jackson , Wei Liu , =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= , Wei Liu Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Stubdomain do not have it's own config file - its configuration is derived from target domains. Do not try to manipulate it when attaching PCI device. This bug prevented starting HVM with stubdomain and PCI passthrough device attached. Signed-off-by: Marek Marczykowski-Górecki Acked-by: Wei Liu --- Changes in v3: - skip libxl__dm_check_start too, as stubdomain is guaranteed to be running at this stage already - do not init d_config at all, as it is used only for json manipulation Changes in v4: - adjust comment style --- tools/libxl/libxl_pci.c | 50 ++++++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 18089ea..9144815 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -120,10 +120,14 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d libxl_domain_config d_config; libxl_device_pci pcidev_saved; libxl__domain_userdata_lock *lock = NULL; + bool is_stubdomain = libxl_is_stubdom(CTX, domid, NULL); - libxl_domain_config_init(&d_config); - libxl_device_pci_init(&pcidev_saved); - libxl_device_pci_copy(CTX, &pcidev_saved, pcidev); + /* Stubdomain doesn't have own config. */ + if (!is_stubdomain) { + libxl_domain_config_init(&d_config); + libxl_device_pci_init(&pcidev_saved); + libxl_device_pci_copy(CTX, &pcidev_saved, pcidev); + } be_path = libxl__domain_device_backend_path(gc, 0, domid, 0, LIBXL__DEVICE_KIND_PCI); @@ -152,27 +156,35 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d GCNEW(device); libxl__device_from_pcidev(gc, domid, pcidev, device); - lock = libxl__lock_domain_userdata(gc, domid); - if (!lock) { - rc = ERROR_LOCK_FAIL; - goto out; - } + /* + * Stubdomin config is derived from its target domain, it doesn't have + * its own file. + */ + if (!is_stubdomain) { + lock = libxl__lock_domain_userdata(gc, domid); + if (!lock) { + rc = ERROR_LOCK_FAIL; + goto out; + } - rc = libxl__get_domain_configuration(gc, domid, &d_config); - if (rc) goto out; + rc = libxl__get_domain_configuration(gc, domid, &d_config); + if (rc) goto out; - device_add_domain_config(gc, &d_config, &libxl__pcidev_devtype, - &pcidev_saved); + device_add_domain_config(gc, &d_config, &libxl__pcidev_devtype, + &pcidev_saved); - rc = libxl__dm_check_start(gc, &d_config, domid); - if (rc) goto out; + rc = libxl__dm_check_start(gc, &d_config, domid); + if (rc) goto out; + } for (;;) { rc = libxl__xs_transaction_start(gc, &t); if (rc) goto out; - rc = libxl__set_domain_configuration(gc, domid, &d_config); - if (rc) goto out; + if (lock) { + rc = libxl__set_domain_configuration(gc, domid, &d_config); + if (rc) goto out; + } libxl__xs_writev(gc, t, be_path, libxl__xs_kvs_of_flexarray(gc, back)); @@ -184,8 +196,10 @@ static int libxl__device_pci_add_xenstore(libxl__gc *gc, uint32_t domid, libxl_d out: libxl__xs_transaction_abort(gc, &t); if (lock) libxl__unlock_domain_userdata(lock); - libxl_device_pci_dispose(&pcidev_saved); - libxl_domain_config_dispose(&d_config); + if (!is_stubdomain) { + libxl_device_pci_dispose(&pcidev_saved); + libxl_domain_config_dispose(&d_config); + } return rc; } From patchwork Wed Jul 17 01:00:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 11046861 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F2C16C5 for ; Wed, 17 Jul 2019 01:03:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0FA7A286BC for ; Wed, 17 Jul 2019 01:03:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ECA282872A; Wed, 17 Jul 2019 01:03:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C82EB2871E for ; Wed, 17 Jul 2019 01:03:42 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYK2-00054d-Ic; Wed, 17 Jul 2019 01:01:30 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYK0-00054G-Vr for xen-devel@lists.xenproject.org; Wed, 17 Jul 2019 01:01:29 +0000 X-Inumbo-ID: 6763abf1-a82e-11e9-8980-bc764e045a96 Received: from new3-smtp.messagingengine.com (unknown [66.111.4.229]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id 6763abf1-a82e-11e9-8980-bc764e045a96; Wed, 17 Jul 2019 01:01:27 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailnew.nyi.internal (Postfix) with ESMTP id A45702F83; Tue, 16 Jul 2019 21:01:26 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Tue, 16 Jul 2019 21:01:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=AC2Lao6o/QARAgAoAa4syLbv6oH/XwjaYrpSUTCEq wE=; b=Wyfm842xZ1m+dvPR2cxEIrNuoG+0R/3BR/T+XDRutVjK14jVFe6JHuRZd vVlc7OoF9FNaoilMKaELvtQl8ZMnEgXKYzCmkAUGIj4KXc0/nIidXyJjpF5v1pSd fkwRVcGWOOGYbV27jC6zogdKB9AR3VQ92ky6fYTZlUuLCXUjK/16Hz4yQVynQDJk a+TCnv8Mfu0iJKIMH9gsZDhcxOgdJ9QaTUHe7xa8MUK+vm97mT/MGqx0yYC0CNyA w0fTWTo6lJN+nSlb904iKbbkLGfVeLhweqsdWk46xgRRIKXWvVSvEhOIhHmNRdiL Ia0H01oHMxrS6Za2FFb2Lr/EKDHyg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddriedugdegvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucffohhmrghinhepghhithhhuhgs rdgtohhmnecukfhppeeluddrieehrdefgedrfeefnecurfgrrhgrmhepmhgrihhlfhhroh hmpehmrghrmhgrrhgvkhesihhnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhmnecu vehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from localhost.localdomain (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id 7F6A0380086; Tue, 16 Jul 2019 21:01:24 -0400 (EDT) From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Date: Wed, 17 Jul 2019 03:00:42 +0200 Message-Id: X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v5 4/6] xen/x86: Allow stubdom access to irq created for msi. X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Kevin Tian , Stefano Stabellini , Suravee Suthikulpanit , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , =?utf-8?q?Marek_Marczykowski-G?= =?utf-8?q?=C3=B3recki?= , Tim Deegan , Simon Gaiser , Julien Grall , Jan Beulich , Brian Woods , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Stubdomains need to be given sufficient privilege over the guest which it provides emulation for in order for PCI passthrough to work correctly. When a HVM domain try to enable MSI, QEMU in stubdomain calls PHYSDEVOP_map_pirq, but later it needs to call XEN_DOMCTL_bind_pt_irq as part of xc_domain_update_msi_irq. Allow for that as part of PHYSDEVOP_map_pirq. This is not needed for PCI INTx, because IRQ in that case is known beforehand and the stubdomain is given permissions over this IRQ by libxl__device_pci_add (there's a do_pci_add against the stubdomain). create_irq() already grant IRQ access to hardware_domain, with assumption the device model (something managing this IRQ) lives there. Modify create_irq() to take additional parameter pointing at device model domain - which may be dom0 or stubdomain. Save ID of the domain given permission, to revoke it in destroy_irq() - easier and cleaner than replaying logic of create_irq() parameter. Use domid instead of actual reference to the domain, because it might get destroyed before destroying IRQ (stubdomain is destroyed before its target domain). And it is not an issue, because IRQ permissions live within domain structure, so destroying a domain also implicitly revoke the permission. Potential domid reuse is detected by by checking if that domain does have permission over the IRQ being destroyed. Then, adjust all callers to provide the parameter. In case of calls not related to stubdomain-initiated allocations, give it either hardware_domain (so the behavior is unchanged there), or NULL for interrupts used by Xen internally. Inspired by https://github.com/OpenXT/xenclient-oe/blob/5e0e7304a5a3c75ef01240a1e3673665b2aaf05e/recipes-extended/xen/files/stubdomain-msi-irq-access.patch by Eric Chanudet . Signed-off-by: Simon Gaiser Signed-off-by: Marek Marczykowski-Górecki --- Changes in v3: - extend commit message Changes in v4: - add missing destroy_irq on error path Changes in v5: - move irq_{grant,revoke}_access() to {create,destroy}_irq(), which basically make it a different patch - add get_dm_domain() helper - do not give hardware_domain permission over IRQs used in Xen internally - rename create_irq argument to just 'd', to avoid confusion when it's called by hardware domain - verify that device is de-assigned before pci_remove_device call - save ID of domain given permission in create_irq(), to revoke it in destroy_irq() - drop domain parameter from destroy_irq() and msi_free_irq() - do not give hardware domain permission over IRQ created in iommu_set_interrupt() --- xen/arch/x86/hpet.c | 3 +- xen/arch/x86/irq.c | 51 ++++++++++++++++++------- xen/common/irq.c | 1 +- xen/drivers/char/ns16550.c | 2 +- xen/drivers/passthrough/amd/iommu_init.c | 2 +- xen/drivers/passthrough/pci.c | 3 +- xen/drivers/passthrough/vtd/iommu.c | 3 +- xen/include/asm-x86/irq.h | 2 +- xen/include/xen/irq.h | 1 +- 9 files changed, 50 insertions(+), 18 deletions(-) diff --git a/xen/arch/x86/hpet.c b/xen/arch/x86/hpet.c index 4b08488..b4854ff 100644 --- a/xen/arch/x86/hpet.c +++ b/xen/arch/x86/hpet.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include #include @@ -368,7 +369,7 @@ static int __init hpet_assign_irq(struct hpet_event_channel *ch) { int irq; - if ( (irq = create_irq(NUMA_NO_NODE)) < 0 ) + if ( (irq = create_irq(NUMA_NO_NODE, hardware_domain)) < 0 ) return irq; ch->msi.irq = irq; diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c index 2cac28a..dc5d302 100644 --- a/xen/arch/x86/irq.c +++ b/xen/arch/x86/irq.c @@ -164,10 +164,21 @@ int __init bind_irq_vector(int irq, int vector, const cpumask_t *cpu_mask) return ret; } +static struct domain *get_dm_domain(struct domain *d) +{ + return current->domain->target == d ? current->domain : hardware_domain; +} + /* * Dynamic irq allocate and deallocation for MSI */ -int create_irq(nodeid_t node) + +/* + * create_irq - allocate irq for MSI + * @d domain that will get permission over the allocated irq; this permission + * will automatically be revoked on destroy_irq + */ +int create_irq(nodeid_t node, struct domain *d) { int irq, ret; struct irq_desc *desc; @@ -200,18 +211,24 @@ int create_irq(nodeid_t node) desc->arch.used = IRQ_UNUSED; irq = ret; } - else if ( hardware_domain ) + else if ( d ) { - ret = irq_permit_access(hardware_domain, irq); + ASSERT(d == current->domain); + ret = irq_permit_access(d, irq); if ( ret ) printk(XENLOG_G_ERR - "Could not grant Dom0 access to IRQ%d (error %d)\n", - irq, ret); + "Could not grant Dom%u access to IRQ%d (error %d)\n", + d->domain_id, irq, ret); + else + desc->creator_domid = d->domain_id; } return irq; } +/* + * destroy_irq - deallocate irq for MSI + */ void destroy_irq(unsigned int irq) { struct irq_desc *desc = irq_to_desc(irq); @@ -220,14 +237,22 @@ void destroy_irq(unsigned int irq) BUG_ON(!MSI_IRQ(irq)); - if ( hardware_domain ) + if ( desc->creator_domid != DOMID_INVALID ) { - int err = irq_deny_access(hardware_domain, irq); + struct domain *d = get_domain_by_id(desc->creator_domid); - if ( err ) - printk(XENLOG_G_ERR - "Could not revoke Dom0 access to IRQ%u (error %d)\n", - irq, err); + if ( d && irq_access_permitted(d, irq) ) { + int err; + + err = irq_deny_access(d, irq); + if ( err ) + printk(XENLOG_G_ERR + "Could not revoke Dom%u access to IRQ%u (error %d)\n", + d->domain_id, irq, err); + } + + if ( d ) + put_domain(d); } spin_lock_irqsave(&desc->lock, flags); @@ -2058,7 +2083,7 @@ int map_domain_pirq( spin_unlock_irqrestore(&desc->lock, flags); info = NULL; - irq = create_irq(NUMA_NO_NODE); + irq = create_irq(NUMA_NO_NODE, get_dm_domain(d)); ret = irq >= 0 ? prepare_domain_irq_pirq(d, irq, pirq + nr, &info) : irq; if ( ret < 0 ) @@ -2691,7 +2716,7 @@ int allocate_and_map_msi_pirq(struct domain *d, int index, int *pirq_p, if ( irq == -1 ) { case MAP_PIRQ_TYPE_MULTI_MSI: - irq = create_irq(NUMA_NO_NODE); + irq = create_irq(NUMA_NO_NODE, get_dm_domain(d)); } if ( irq < nr_irqs_gsi || irq >= nr_irqs ) diff --git a/xen/common/irq.c b/xen/common/irq.c index f42512d..42b27a9 100644 --- a/xen/common/irq.c +++ b/xen/common/irq.c @@ -16,6 +16,7 @@ int init_one_irq_desc(struct irq_desc *desc) spin_lock_init(&desc->lock); cpumask_setall(desc->affinity); INIT_LIST_HEAD(&desc->rl_link); + desc->creator_domid = DOMID_INVALID; err = arch_init_one_irq_desc(desc); if ( err ) diff --git a/xen/drivers/char/ns16550.c b/xen/drivers/char/ns16550.c index 189e121..ccc8b04 100644 --- a/xen/drivers/char/ns16550.c +++ b/xen/drivers/char/ns16550.c @@ -719,7 +719,7 @@ static void __init ns16550_init_irq(struct serial_port *port) struct ns16550 *uart = port->uart; if ( uart->msi ) - uart->irq = create_irq(0); + uart->irq = create_irq(0, NULL); #endif } diff --git a/xen/drivers/passthrough/amd/iommu_init.c b/xen/drivers/passthrough/amd/iommu_init.c index 4e76b26..50785e0 100644 --- a/xen/drivers/passthrough/amd/iommu_init.c +++ b/xen/drivers/passthrough/amd/iommu_init.c @@ -781,7 +781,7 @@ static bool_t __init set_iommu_interrupt_handler(struct amd_iommu *iommu) hw_irq_controller *handler; u16 control; - irq = create_irq(NUMA_NO_NODE); + irq = create_irq(NUMA_NO_NODE, NULL); if ( irq <= 0 ) { dprintk(XENLOG_ERR, "IOMMU: no irqs\n"); diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index e886894..507b3d1 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -845,6 +845,9 @@ int pci_remove_device(u16 seg, u8 bus, u8 devfn) list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) if ( pdev->bus == bus && pdev->devfn == devfn ) { + ret = -EBUSY; + if ( pdev->domain && pdev->domain != hardware_domain ) + break; ret = iommu_remove_device(pdev); if ( pdev->domain ) list_del(&pdev->domain_list); diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c index 8b27d7e..79f9682 100644 --- a/xen/drivers/passthrough/vtd/iommu.c +++ b/xen/drivers/passthrough/vtd/iommu.c @@ -1138,7 +1138,8 @@ static int __init iommu_set_interrupt(struct acpi_drhd_unit *drhd) struct irq_desc *desc; irq = create_irq(rhsa ? pxm_to_node(rhsa->proximity_domain) - : NUMA_NO_NODE); + : NUMA_NO_NODE, + NULL); if ( irq <= 0 ) { dprintk(XENLOG_ERR VTDPREFIX, "IOMMU: no irq available!\n"); diff --git a/xen/include/asm-x86/irq.h b/xen/include/asm-x86/irq.h index c0c6e7c..5b24428 100644 --- a/xen/include/asm-x86/irq.h +++ b/xen/include/asm-x86/irq.h @@ -155,7 +155,7 @@ int init_irq_data(void); void clear_irq_vector(int irq); int irq_to_vector(int irq); -int create_irq(nodeid_t node); +int create_irq(nodeid_t node, struct domain *d); void destroy_irq(unsigned int irq); int assign_irq_vector(int irq, const cpumask_t *); diff --git a/xen/include/xen/irq.h b/xen/include/xen/irq.h index 586b783..c7a6a83 100644 --- a/xen/include/xen/irq.h +++ b/xen/include/xen/irq.h @@ -91,6 +91,7 @@ typedef struct irq_desc { spinlock_t lock; struct arch_irq_desc arch; cpumask_var_t affinity; + domid_t creator_domid; /* weak reference to domain handling this IRQ */ /* irq ratelimit */ s_time_t rl_quantum_start; From patchwork Wed Jul 17 01:00:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 11046857 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 97C076C5 for ; Wed, 17 Jul 2019 01:03:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 87DAF286C6 for ; Wed, 17 Jul 2019 01:03:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 74F4728734; Wed, 17 Jul 2019 01:03:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 53537285ED for ; Wed, 17 Jul 2019 01:03:16 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYK3-000554-Um; Wed, 17 Jul 2019 01:01:31 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYK3-00054t-57 for xen-devel@lists.xenproject.org; Wed, 17 Jul 2019 01:01:31 +0000 X-Inumbo-ID: 68650566-a82e-11e9-ac55-63be4eb0056e Received: from new3-smtp.messagingengine.com (unknown [66.111.4.229]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 68650566-a82e-11e9-ac55-63be4eb0056e; Wed, 17 Jul 2019 01:01:28 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailnew.nyi.internal (Postfix) with ESMTP id 678DD2818; Tue, 16 Jul 2019 21:01:28 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Tue, 16 Jul 2019 21:01:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=oXw5SYpgWONCsQMlX8Tt0b6R78vNuA4o5rHtiIe0f 38=; b=dzcZv2sh31pMfqrHX68hT8O0/s/4lkmZTBmgyf45IiQPtWLtfLr6qFzwk viO0bVYCX+T6hCB77CaO0dr+bK4TjvQq/pW1IiA+dgZHVEdxXZZ4i2BN1wmExxtv P6WVlxkB8xXnHWRjT/y4ovhenOUIFAo8pHuGsdJ4kAZuJdroiUOxMU9CJNysnMtg Ih6myif9KbTwybJb84E04kmkaj7AIRKgZ8QLow9JtdbqdDftCQDoka8Kc7VQh1VI Zz9HJX86hDcy/QkMovjYsQmTz3YwYbK1AjZ1Wo2ExXM+vckDflPYCX3mjMP+RjZS /eMiur+2BD/XShLIgLW9Y96ugmZKQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddriedugdegvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucfkphepledurdeihedrfeegrdef feenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghksehinhhvihhsihgslh gvthhhihhnghhslhgrsgdrtghomhenucevlhhushhtvghrufhiiigvpedu X-ME-Proxy: Received: from localhost.localdomain (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id 923C638008A; Tue, 16 Jul 2019 21:01:26 -0400 (EDT) From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Date: Wed, 17 Jul 2019 03:00:43 +0200 Message-Id: <7280f16277aa9d7bdc9c2373277ef1b40459090b.1563325215.git-series.marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v5 5/6] xen/x86: add PHYSDEVOP_msi_control X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Stefano Stabellini , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , =?utf-8?q?Marek_Marczykowski-G?= =?utf-8?q?=C3=B3recki?= , Tim Deegan , Julien Grall , Jan Beulich , Daniel De Graaf , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Allow device model running in stubdomain to enable/disable MSI(-X), bypassing pciback. While pciback is still used to access config space from within stubdomain, it refuse to write to PCI_MSI_FLAGS_ENABLE/PCI_MSIX_FLAGS_ENABLE in non-permissive mode. Which is the right thing to do for PV domain (the main use case for pciback), as PV domain should use XEN_PCI_OP_* commands for that. Unfortunately those commands are not good for stubdomain use, as they configure MSI in dom0's kernel too, which should not happen for HVM domain. This new physdevop is allowed only for stubdomain controlling the domain which own the device. Signed-off-by: Marek Marczykowski-Górecki --- Changes in v3: - new patch Changes in v4: - adjust code style - s/msi_msix/msi/ - add msi_set_enable XSM hook - flatten struct physdev_msi_set_enable - add to include/xlat.lst Changes in v5: - rename to PHYSDEVOP_msi_control - combine "mode" and "enable" into "flags" - refuse to enable both MSI and MSI-X, and also to enable MSI(-X) on incapable device - disable/enable INTx when enabling/disabling MSI (?) - refuse if !use_msi - adjust flask hook to make more sense (require "setup" access on device, not on domain) - rebase on master I'm not sure if XSM part is correct, compile-tested only, as I'm not sure how to set the policy. --- xen/arch/x86/msi.c | 42 ++++++++++++++++++++++++++++++- xen/arch/x86/physdev.c | 25 ++++++++++++++++++- xen/arch/x86/x86_64/physdev.c | 4 +++- xen/include/asm-x86/msi.h | 1 +- xen/include/public/physdev.h | 16 +++++++++++- xen/include/xlat.lst | 1 +- xen/include/xsm/dummy.h | 7 +++++- xen/include/xsm/xsm.h | 6 ++++- xen/xsm/dummy.c | 1 +- xen/xsm/flask/hooks.c | 24 +++++++++++++++++- xen/xsm/flask/policy/access_vectors | 1 +- 11 files changed, 128 insertions(+) diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c index 89e6116..fca1d04 100644 --- a/xen/arch/x86/msi.c +++ b/xen/arch/x86/msi.c @@ -1475,6 +1475,48 @@ int pci_restore_msi_state(struct pci_dev *pdev) return 0; } +int msi_control(struct pci_dev *pdev, bool msix, bool enable) +{ + int ret; + struct msi_desc *old_desc; + + if ( !use_msi ) + return -EOPNOTSUPP; + + ret = xsm_msi_control(XSM_DM_PRIV, pdev->domain, pdev->sbdf.sbdf, msix, enable); + if ( ret ) + return ret; + + if ( msix ) + { + if ( !pdev->msix ) + return -ENODEV; + old_desc = find_msi_entry(pdev, -1, PCI_CAP_ID_MSI); + if ( old_desc ) + return -EBUSY; + if ( enable ) + pci_intx(pdev, false); + msix_set_enable(pdev, enable); + } + else + { + if ( !pci_find_cap_offset(pdev->seg, + pdev->bus, + PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn), + PCI_CAP_ID_MSI) ) + return -ENODEV; + old_desc = find_msi_entry(pdev, -1, PCI_CAP_ID_MSIX); + if ( old_desc ) + return -EBUSY; + if ( enable ) + pci_intx(pdev, false); + msi_set_enable(pdev, enable); + } + + return 0; +} + void __init early_msi_init(void) { if ( use_msi < 0 ) diff --git a/xen/arch/x86/physdev.c b/xen/arch/x86/physdev.c index 3a3c158..5000998 100644 --- a/xen/arch/x86/physdev.c +++ b/xen/arch/x86/physdev.c @@ -662,6 +662,31 @@ ret_t do_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) break; } + case PHYSDEVOP_msi_control: { + struct physdev_msi_control op; + struct pci_dev *pdev; + + ret = -EFAULT; + if ( copy_from_guest(&op, arg, 1) ) + break; + + ret = -EINVAL; + if ( op.flags & ~(PHYSDEVOP_MSI_CONTROL_MSIX | PHYSDEVOP_MSI_CONTROL_ENABLE) ) + break; + + pcidevs_lock(); + pdev = pci_get_pdev(op.seg, op.bus, op.devfn); + if ( pdev ) + ret = msi_control(pdev, + op.flags & PHYSDEVOP_MSI_CONTROL_MSIX, + op.flags & PHYSDEVOP_MSI_CONTROL_ENABLE); + else + ret = -ENODEV; + pcidevs_unlock(); + break; + + } + default: ret = -ENOSYS; break; diff --git a/xen/arch/x86/x86_64/physdev.c b/xen/arch/x86/x86_64/physdev.c index c5a00ea..69b4ce3 100644 --- a/xen/arch/x86/x86_64/physdev.c +++ b/xen/arch/x86/x86_64/physdev.c @@ -76,6 +76,10 @@ CHECK_physdev_pci_device_add CHECK_physdev_pci_device #undef xen_physdev_pci_device +#define xen_physdev_msi_control physdev_msi_control +CHECK_physdev_msi_control +#undef xen_physdev_msi_control + #define COMPAT #undef guest_handle_okay #define guest_handle_okay compat_handle_okay diff --git a/xen/include/asm-x86/msi.h b/xen/include/asm-x86/msi.h index 10387dc..05296de 100644 --- a/xen/include/asm-x86/msi.h +++ b/xen/include/asm-x86/msi.h @@ -252,5 +252,6 @@ void guest_mask_msi_irq(struct irq_desc *, bool mask); void ack_nonmaskable_msi_irq(struct irq_desc *); void end_nonmaskable_msi_irq(struct irq_desc *, u8 vector); void set_msi_affinity(struct irq_desc *, const cpumask_t *); +int msi_control(struct pci_dev *pdev, bool msix, bool enable); #endif /* __ASM_MSI_H */ diff --git a/xen/include/public/physdev.h b/xen/include/public/physdev.h index b6faf83..f9b728f 100644 --- a/xen/include/public/physdev.h +++ b/xen/include/public/physdev.h @@ -344,6 +344,22 @@ struct physdev_dbgp_op { typedef struct physdev_dbgp_op physdev_dbgp_op_t; DEFINE_XEN_GUEST_HANDLE(physdev_dbgp_op_t); +/* when PHYSDEVOP_MSI_CONTROL_MSIX not set, control MSI */ +#define PHYSDEVOP_MSI_CONTROL_MSIX 1 +/* when PHYSDEVOP_MSI_CONTROL_ENABLE not set, disable */ +#define PHYSDEVOP_MSI_CONTROL_ENABLE 2 + +#define PHYSDEVOP_msi_control 32 +struct physdev_msi_control { + /* IN */ + uint16_t seg; + uint8_t bus; + uint8_t devfn; + uint8_t flags; +}; +typedef struct physdev_msi_control physdev_msi_control_t; +DEFINE_XEN_GUEST_HANDLE(physdev_msi_control_t); + /* * Notify that some PIRQ-bound event channels have been unmasked. * ** This command is obsolete since interface version 0x00030202 and is ** diff --git a/xen/include/xlat.lst b/xen/include/xlat.lst index 95f5e55..3082761 100644 --- a/xen/include/xlat.lst +++ b/xen/include/xlat.lst @@ -108,6 +108,7 @@ ? physdev_irq_status_query physdev.h ? physdev_manage_pci physdev.h ? physdev_manage_pci_ext physdev.h +? physdev_msi_control physdev.h ? physdev_pci_device physdev.h ? physdev_pci_device_add physdev.h ? physdev_pci_mmcfg_reserved physdev.h diff --git a/xen/include/xsm/dummy.h b/xen/include/xsm/dummy.h index 01d2814..4801838 100644 --- a/xen/include/xsm/dummy.h +++ b/xen/include/xsm/dummy.h @@ -514,6 +514,13 @@ static XSM_INLINE int xsm_pci_config_permission(XSM_DEFAULT_ARG struct domain *d return xsm_default_action(action, current->domain, d); } +static XSM_INLINE int xsm_msi_control(XSM_DEFAULT_ARG struct domain *d, uint32_t machine_bdf, + uint8_t msix, uint8_t enable) +{ + XSM_ASSERT_ACTION(XSM_DM_PRIV); + return xsm_default_action(action, current->domain, d); +} + static XSM_INLINE int xsm_add_to_physmap(XSM_DEFAULT_ARG struct domain *d1, struct domain *d2) { XSM_ASSERT_ACTION(XSM_TARGET); diff --git a/xen/include/xsm/xsm.h b/xen/include/xsm/xsm.h index b6141f6..bf39dbd 100644 --- a/xen/include/xsm/xsm.h +++ b/xen/include/xsm/xsm.h @@ -106,6 +106,7 @@ struct xsm_operations { int (*iomem_permission) (struct domain *d, uint64_t s, uint64_t e, uint8_t allow); int (*iomem_mapping) (struct domain *d, uint64_t s, uint64_t e, uint8_t allow); int (*pci_config_permission) (struct domain *d, uint32_t machine_bdf, uint16_t start, uint16_t end, uint8_t access); + int (*msi_control) (struct domain *d, uint32_t machine_bdf, uint8_t msix, uint8_t enable); #if defined(CONFIG_HAS_PASSTHROUGH) && defined(CONFIG_HAS_PCI) int (*get_device_group) (uint32_t machine_bdf); @@ -464,6 +465,11 @@ static inline int xsm_pci_config_permission (xsm_default_t def, struct domain *d return xsm_ops->pci_config_permission(d, machine_bdf, start, end, access); } +static inline int xsm_msi_control (xsm_default_t def, struct domain *d, uint32_t machine_bdf, uint8_t msix, uint8_t enable) +{ + return xsm_ops->msi_control(d, machine_bdf, msix, enable); +} + #if defined(CONFIG_HAS_PASSTHROUGH) && defined(CONFIG_HAS_PCI) static inline int xsm_get_device_group(xsm_default_t def, uint32_t machine_bdf) { diff --git a/xen/xsm/dummy.c b/xen/xsm/dummy.c index c9a566f..878eefe 100644 --- a/xen/xsm/dummy.c +++ b/xen/xsm/dummy.c @@ -81,6 +81,7 @@ void __init xsm_fixup_ops (struct xsm_operations *ops) set_to_dummy_if_null(ops, iomem_permission); set_to_dummy_if_null(ops, iomem_mapping); set_to_dummy_if_null(ops, pci_config_permission); + set_to_dummy_if_null(ops, msi_control); set_to_dummy_if_null(ops, get_vnumainfo); #if defined(CONFIG_HAS_PASSTHROUGH) && defined(CONFIG_HAS_PCI) diff --git a/xen/xsm/flask/hooks.c b/xen/xsm/flask/hooks.c index a7d690a..5fb755e 100644 --- a/xen/xsm/flask/hooks.c +++ b/xen/xsm/flask/hooks.c @@ -1083,6 +1083,29 @@ static int flask_pci_config_permission(struct domain *d, uint32_t machine_bdf, u } +static int flask_msi_control(struct domain *d, uint32_t machine_bdf, uint8_t msix, uint8_t enable) +{ + uint32_t dsid, rsid; + int rc = -EPERM; + struct avc_audit_data ad; + uint32_t perm; + + AVC_AUDIT_DATA_INIT(&ad, DEV); + ad.device = machine_bdf; + + rc = security_device_sid(machine_bdf, &rsid); + if ( rc ) + return rc; + + rc = avc_current_has_perm(rsid, SECCLASS_RESOURCE, RESOURCE__SETUP, &ad); + if ( rc ) + return rc; + + perm = flask_iommu_resource_use_perm(); + dsid = domain_sid(d); + return avc_has_perm(dsid, rsid, SECCLASS_RESOURCE, perm, &ad); +} + static int flask_resource_plug_core(void) { return avc_current_has_perm(SECINITSID_DOMXEN, SECCLASS_RESOURCE, RESOURCE__PLUG, NULL); @@ -1800,6 +1823,7 @@ static struct xsm_operations flask_ops = { .iomem_permission = flask_iomem_permission, .iomem_mapping = flask_iomem_mapping, .pci_config_permission = flask_pci_config_permission, + .msi_control = flask_msi_control, .resource_plug_core = flask_resource_plug_core, .resource_unplug_core = flask_resource_unplug_core, diff --git a/xen/xsm/flask/policy/access_vectors b/xen/xsm/flask/policy/access_vectors index 194d743..0ddfc91 100644 --- a/xen/xsm/flask/policy/access_vectors +++ b/xen/xsm/flask/policy/access_vectors @@ -466,6 +466,7 @@ class resource # checked for PHYSDEVOP_restore_msi* (target PCI device) # checked for PHYSDEVOP_setup_gsi (target IRQ) # checked for PHYSDEVOP_pci_mmcfg_reserved (target xen_t) +# checked for PHYSDEVOP_msi_control (target PCI device) setup } From patchwork Wed Jul 17 01:00:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 11046859 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 35E646C5 for ; Wed, 17 Jul 2019 01:03:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 25313286C6 for ; Wed, 17 Jul 2019 01:03:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 193BA286E2; Wed, 17 Jul 2019 01:03:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B1FA02872A for ; Wed, 17 Jul 2019 01:03:35 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYK6-00055Z-Bc; Wed, 17 Jul 2019 01:01:34 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnYK4-00055O-Fp for xen-devel@lists.xenproject.org; Wed, 17 Jul 2019 01:01:32 +0000 X-Inumbo-ID: 695ccef4-a82e-11e9-aa66-8bb310a9bfcd Received: from wout4-smtp.messagingengine.com (unknown [64.147.123.20]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 695ccef4-a82e-11e9-aa66-8bb310a9bfcd; Wed, 17 Jul 2019 01:01:30 +0000 (UTC) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id 8BD4D3EB; Tue, 16 Jul 2019 21:01:29 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Tue, 16 Jul 2019 21:01:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=SnT1nCoHjJFRhs+EcWdbFq47ZHj0OL2KgdeM1Eytu WI=; b=opmNLfG8MI8BJSyYEkKxko6Hfo2o9oPCMhQYhd1Ev+mGCbmufZABZ/fz9 IYu15kM2Bpej9VPF//OAHaz/J4unp0lwLhgibvcY6CK2DwPEC+5kHNgTJxbLAEXK CH1P9Lgdzqog+U0ndMoKb3mTUubCWkneD2jXnCTv4JcSJKx0+6drjWfBIwboPxI1 xbQDIojXMcuU46NCTj67NndZ37vaLcDkHewxWwog/h94IWKN8Z8+EgCiWACIhUWY 9Ygsb2Kfa5YXI8msaUqvT/wV1jaP4lwYstmAb/xWNZrUUqeX2MxXI/OKjh9e2423 8Ct3q+w+Xybc3+0xxIylAMQ0de+PA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduvddriedugdegvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhggtgfgsehtkeertdertdejnecuhfhrohhmpeforghrvghk ucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmhgrrhgvkhesihhnvh hishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucfkphepledurdeihedrfeegrdef feenucfrrghrrghmpehmrghilhhfrhhomhepmhgrrhhmrghrvghksehinhhvihhsihgslh gvthhhihhnghhslhgrsgdrtghomhenucevlhhushhtvghrufhiiigvpedu X-ME-Proxy: Received: from localhost.localdomain (ip5b412221.dynamic.kabel-deutschland.de [91.65.34.33]) by mail.messagingengine.com (Postfix) with ESMTPA id 5302F38008D; Tue, 16 Jul 2019 21:01:28 -0400 (EDT) From: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= To: xen-devel@lists.xenproject.org Date: Wed, 17 Jul 2019 03:00:44 +0200 Message-Id: <5c8fb8141c0f92ccaa78cdc169b8544f7634ce65.1563325215.git-series.marmarek@invisiblethingslab.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: References: MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v5 6/6] tools/libxc: add wrapper for PHYSDEVOP_msi_control X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , =?utf-8?q?Marek_Marczykowski-G?= =?utf-8?q?=C3=B3recki?= , Wei Liu Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Add libxc wrapper for PHYSDEVOP_msi_control introduced in previous commit. Signed-off-by: Marek Marczykowski-Górecki Reviewed-by: Roger Pau Monné --- Changes in v3: - new patch Changes in v4: - adjust for updated previous patch Changes in v5: - rename to PHYSDEVOP_msi_control, adjust arguments --- tools/libxc/include/xenctrl.h | 6 ++++++ tools/libxc/xc_physdev.c | 19 +++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h index 538007a..826d10d 100644 --- a/tools/libxc/include/xenctrl.h +++ b/tools/libxc/include/xenctrl.h @@ -1638,6 +1638,12 @@ int xc_physdev_unmap_pirq(xc_interface *xch, uint32_t domid, int pirq); +int xc_physdev_msi_control(xc_interface *xch, + int seg, + int bus, + int devfn, + int flags); + /* * LOGGING AND ERROR REPORTING */ diff --git a/tools/libxc/xc_physdev.c b/tools/libxc/xc_physdev.c index 460a8e7..a25a117 100644 --- a/tools/libxc/xc_physdev.c +++ b/tools/libxc/xc_physdev.c @@ -111,3 +111,22 @@ int xc_physdev_unmap_pirq(xc_interface *xch, return rc; } +int xc_physdev_msi_control(xc_interface *xch, + int seg, + int bus, + int devfn, + int flags) +{ + int rc; + struct physdev_msi_control op; + + memset(&op, 0, sizeof(struct physdev_msi_control)); + op.seg = seg; + op.bus = bus; + op.devfn = devfn; + op.flags = flags; + + rc = do_physdev_op(xch, PHYSDEVOP_msi_control, &op, sizeof(op)); + + return rc; +}