From patchwork Fri Sep 23 14:25:41 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Marek_Marczykowski-G=C3=B3recki?= X-Patchwork-Id: 9348217 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D91336077A for ; Fri, 23 Sep 2016 14:28:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C7D942ABDA for ; Fri, 23 Sep 2016 14:28:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BC9012ABEF; Fri, 23 Sep 2016 14:28:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B44692ABED for ; Fri, 23 Sep 2016 14:28:02 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bnRQG-0008L0-N7; Fri, 23 Sep 2016 14:25:52 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bnRQF-0008KN-Ej for xen-devel@lists.xen.org; Fri, 23 Sep 2016 14:25:51 +0000 Received: from [85.158.139.211] by server-17.bemta-5.messagelabs.com id 53/EA-03778-E6B35E75; Fri, 23 Sep 2016 14:25:50 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOKsWRWlGSWpSXmKPExsXilM8ipZtj/TT cYN8eOYslHxezODB6HN39mymAMYo1My8pvyKBNWPu/YiCbT4Vxx4GNzDet+1i5OIQEljFKLHi SjdbFyMnkJMt0TFrKwuIzSIwl1liw44EkCIWgdnMEp+O3QUrkhBwlpi/dhYTRMMGRon+Tw5dj BxARaoSF5dWgYTZBIIlri/5xQpiiwjkS0yfOA+sXFjAQuLI4iZmEJtXQFeicV0jI8SYGImtTR Og4oISJ2c+AbuBWaBU4veiDUwg45kFpCWW/+MACXMKWEt0/jnGDmKLCihLNMx4wDyBUXAWku5 ZSLpnIXRDhK0k5nVNZMEQVpf4M+8SM4awtsSyha+ZUZWA2EkSM+c2suNXky1xpHc5E6YaW4l1 696zLGDkX8WoXpxaVJZapGuql1SUmZ5RkpuYmaNraGCql5taXJyYnpqTmFSsl5yfu4kRGMkMQ LCD8Uu/8yFGSQ4mJVHexn1PwoX4kvJTKjMSizPii0pzUosPMcpwcChJ8DoYPA0XEixKTU+tSM vMAaYUmLQEB4+SCG+yFVCat7ggMbc4Mx0idYpRl2PLghtrmYRY8vLzUqXEeeVBigRAijJK8+B GwNLbJUZZKWFeRqCjhHgKUotyM0tQ5V8xinMwKgnz+oNM4cnMK4Hb9AroCCagI77deQJyREki QkqqgdEqsotvTdTe7mdLb73al7frg8fjjGdnw2pvT1wne8zzYtn/LS+vpf04cfZK61+OZTdFv Zb0OCxJVbilfy7X7erXaSZFn2+XpfRs5Mn5UuHQxhOecHfTr9iy36eyAn3qU3d/tZQz8g236J 3XuLTgfK90YvosG39m428yAvuOCuXUlDF/ctBk9FViKc5INNRiLipOBABCS038agMAAA== X-Env-Sender: marmarek@invisiblethingslab.com X-Msg-Ref: server-15.tower-206.messagelabs.com!1474640747!47404285!1 X-Originating-IP: [66.111.4.26] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTExLjQuMjYgPT4gMTIyNTM=\n X-StarScan-Received: X-StarScan-Version: 8.84; banners=-,-,- X-VirusChecked: Checked Received: (qmail 54220 invoked from network); 23 Sep 2016 14:25:48 -0000 Received: from out2-smtp.messagingengine.com (HELO out2-smtp.messagingengine.com) (66.111.4.26) by server-15.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 23 Sep 2016 14:25:48 -0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id 5FC5420749; Fri, 23 Sep 2016 10:25:47 -0400 (EDT) Received: from frontend2 ([10.202.2.161]) by compute2.internal (MEProxy); Fri, 23 Sep 2016 10:25:47 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= invisiblethingslab.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-sasl-enc :x-sasl-enc; s=mesmtp; bh=FVLOUfjbof9V67OadbdkXO/hr2Q=; b=TI9LLF tdbNekquWcmleY4jsgQ3jH5ciuCSlPmblJ2wIZ5vNDZwNr+dEZJFbsng3G0OtgIV +FzA0B5jqMbkQvnRvX2tp1dkmzhiHwwgXu0dJQ1c3IJjyR/lirakO0UXS+crzip+ d0LV0XQxcDVMAGeqkpBnFbMm+kK6sBu8WBVms= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-sasl-enc :x-sasl-enc; s=smtpout; bh=FVLOUfjbof9V67OadbdkXO/hr2Q=; b=bRi7G pC/FRXdZzKBgTm7LJh1lpQBhXTl2ZXeZCUoM+lOQBz7Lw4FduIVvwBvQaB5nj59y ilgOidBeONByC4y8scAIyb7DoWmts6RoFYo/oDP74S7vtI3+wNNueDy66FcJ1K05 p0fxJ5iSlrG5OqMKJN+7RgI0cc/YHONquKvP0g= X-Sasl-enc: 1570OLP1rxE5bTy5fceMEjs8xiJWCnVyN0PdIj5BMBC8 1474640746 Received: from mail-itl (89-70-103-23.dynamic.chello.pl [89.70.103.23]) by mail.messagingengine.com (Postfix) with ESMTPA id 71E22CCEA2; Fri, 23 Sep 2016 10:25:46 -0400 (EDT) Date: Fri, 23 Sep 2016 16:25:41 +0200 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: Samuel Thibault , xen-devel , HW42 Message-ID: <20160923142541.GB31510@mail-itl> References: <20160923084814.GS7339@mail-itl> <20160923132707.GH20069@var.bordeaux.inria.fr> MIME-Version: 1.0 In-Reply-To: <20160923132707.GH20069@var.bordeaux.inria.fr> User-Agent: Mutt/1.6.1 (2016-04-27) Subject: Re: [Xen-devel] PCI passthrough with stubdomain X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP On Fri, Sep 23, 2016 at 03:27:07PM +0200, Samuel Thibault wrote: > Marek Marczykowski-Górecki, on Fri 23 Sep 2016 10:48:14 +0200, wrote: > > I'm still trying to get PCI passthrough working with stubdomain and > > without qemu in dom0 (even for just vfb/vkbd backends). How is this > > supposed to work? > > Just as I recall from my memory: > > > 1. Should xen-pcifront in the target domain be used (and consequently, > > should xen-pciback be set for it)? > > I guess that could work. Could, or should? ;) In the meantime, I've found this in xen-pcifront driver: static int __init pcifront_init(void) { if (!xen_pv_domain() || xen_initial_domain()) return -ENODEV; So, it looks like pcifront is not used in PV domain. > > Currently xen-pciback is set for both > > stubdomain and target domain, which presents a race condition and > > xen-pciback refuses to setup one of them. > > Yes, that's expected, for the reason you say. In the meantime I've tried to modify libxl to not setup pciback for target domain. This, along with other modifications (see below) improved the situation. > * to summarize > ************** > > If running PV drivers in the guest, you set the pciback on the guest, be > it run with stubdom or not. > If running plain drivers in the guest, > * when not using a stubdom you don't need to set a pciback. > * when using a stubdom you need to set a pciback on the stubdom. Thanks for the explanation! > So the unfortunate thing is that when using stubdom, you'd have to set > the pciback either on the guest (to run a PV driver in it), or on the > stubdom (to run a plain driver in the guest, and let mini-os use PV to > let qemu pass the board through) I've tried only on Linux HVM guest and as noted above xen-pcifront doesn't look to be involved. Is it any different on other OSes? As for getting PCI passthrough working with stubdomain, for now I hit those problems: 1. getdomaininfo not allowed from stubdomain (already fixed in master), 2. double setup of pciback (explained above) - for now I've disabled one of them, 3. race condition between pcifront in mini-os and qemu. Regarding the last one: Toolstack during (stub)domain startup setup two things, mostly asynchronously: 1. pciback/pcifront (through standard xenstore entries) 2. instruct qemu to attach device (by writing "pci-ins" to device-model/xxx/command) The later fails if pcifront is not initialized yet. For now I've moved the second step to be really after the first one, including synchronization through libxl__wait_for_backend call. Is it ok? Or maybe it should be done on stubdomain side (handling "pci-ins" should wait on pcifront)? I guess the same will apply to qemu-xen case, or any other stubdomain, so probably better have it in toolstack, right? Work-in-progress patches attached. The result is that lspci inside the guest finally list the device. No idea if it's really working yet. diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c index c6862b8..5625ef2 100644 --- a/tools/libxl/libxl_create.c +++ b/tools/libxl/libxl_create.c @@ -1429,7 +1429,7 @@ static void domcreate_attach_pci(libxl__egc *egc, libxl__multidev *multidev, } } - if (d_config->num_pcidevs > 0) { + if (d_config->num_pcidevs > 0 && d_config->c_info.type == LIBXL_DOMAIN_TYPE_PV) { ret = libxl__create_pci_backend(gc, domid, d_config->pcidevs, d_config->num_pcidevs); if (ret < 0) { diff --git a/tools/libxl/libxl_pci.c b/tools/libxl/libxl_pci.c index 19c597e..2942772 100644 --- a/tools/libxl/libxl_pci.c +++ b/tools/libxl/libxl_pci.c @@ -1001,7 +1001,7 @@ out: } } - if (!starting) + if (!starting && !hvm) rc = libxl__device_pci_add_xenstore(gc, domid, pcidev, starting); else rc = 0;