From patchwork Tue Jul 16 10:16:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Durrant X-Patchwork-Id: 11045841 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7589E138D for ; Tue, 16 Jul 2019 10:18:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 65E9C28451 for ; Tue, 16 Jul 2019 10:18:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 59146285A4; Tue, 16 Jul 2019 10:18:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id ADD9228585 for ; Tue, 16 Jul 2019 10:18:49 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnKWD-0006Hf-FR; Tue, 16 Jul 2019 10:17:09 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnKWC-0006Ha-Kg for xen-devel@lists.xenproject.org; Tue, 16 Jul 2019 10:17:08 +0000 X-Inumbo-ID: dd2fae5c-a7b2-11e9-8980-bc764e045a96 Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id dd2fae5c-a7b2-11e9-8980-bc764e045a96; Tue, 16 Jul 2019 10:17:07 +0000 (UTC) Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=paul.durrant@citrix.com; spf=Pass smtp.mailfrom=Paul.Durrant@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa3.hc3370-68.iphmx.com: no sender authenticity information available from domain of paul.durrant@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="paul.durrant@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa3.hc3370-68.iphmx.com: domain of Paul.Durrant@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="Paul.Durrant@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa3.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: JlitRHSHF4OL6NfGG6j6wVYNETYWU77QXWlziPhjIRqnfhHuWD2zjtDCsPS6K4nk+I0TqxZ0/U hS379upSUAUzl0SmjIE0PKbWqrd4hWbWZBYXcAtULTAmpoaPiwox2/JEOTbgEOaX0U1fmGCzO3 FiOvOZ3pKOFHgqZkwF68Jp6xbdMjaBIvQZ16nlF6TYGgzUM2mUYTJOmdDj0jU3ijf6/9n+sUWY 6+nghmr4BpFxCxIJZd4WqvcBlIxvFxz90+a1GHnx1K+knLNIQXLO0vWsSE/hcf18horLnP42je Hag= X-SBRS: 2.7 X-MesageID: 3028178 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,498,1557201600"; d="scan'208";a="3028178" From: Paul Durrant To: Date: Tue, 16 Jul 2019 11:16:54 +0100 Message-ID: <20190716101657.23327-2-paul.durrant@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190716101657.23327-1-paul.durrant@citrix.com> References: <20190716101657.23327-1-paul.durrant@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 1/4] iommu / x86: move call to scan_pci_devices() out of vendor code X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Kevin Tian , Suravee Suthikulpanit , Wei Liu , Andrew Cooper , Paul Durrant , Jan Beulich , Brian Woods Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP It's not vendor specific so it doesn't really belong there. Scanning the PCI topology also really doesn't have much to do with IOMMU initialization. It doesn't depend on there even being an IOMMU. This patch moves to the call to the beginning of iommu_hardware_setup() but only places it there because the topology information would be otherwise unused. Subsequent patches will actually make use of the PCI topology during (x86) IOMMU initialization. Signed-off-by: Paul Durrant Reviewed-by: "Roger Pau Monné" Reviewed-by: Kevin Tian Acked-by: Jan Beulich --- Cc: Suravee Suthikulpanit Cc: Brian Woods Cc: Kevin Tian Cc: Jan Beulich Cc: Andrew Cooper Cc: Wei Liu v2: - Expanded commit comment. - Moved PCI scan to before IOMMU initialization, rather than after it. --- xen/drivers/passthrough/amd/pci_amd_iommu.c | 3 ++- xen/drivers/passthrough/vtd/iommu.c | 4 ---- xen/drivers/passthrough/x86/iommu.c | 6 ++++++ 3 files changed, 8 insertions(+), 5 deletions(-) diff --git a/xen/drivers/passthrough/amd/pci_amd_iommu.c b/xen/drivers/passthrough/amd/pci_amd_iommu.c index 4afbcd1609..3338a8e0e8 100644 --- a/xen/drivers/passthrough/amd/pci_amd_iommu.c +++ b/xen/drivers/passthrough/amd/pci_amd_iommu.c @@ -180,7 +180,8 @@ static int __init iov_detect(void) if ( !amd_iommu_perdev_intremap ) printk(XENLOG_WARNING "AMD-Vi: Using global interrupt remap table is not recommended (see XSA-36)!\n"); - return scan_pci_devices(); + + return 0; } int amd_iommu_alloc_root(struct domain_iommu *hd) diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c index 8b27d7e775..b0e3bf26b5 100644 --- a/xen/drivers/passthrough/vtd/iommu.c +++ b/xen/drivers/passthrough/vtd/iommu.c @@ -2372,10 +2372,6 @@ static int __init vtd_setup(void) P(iommu_hap_pt_share, "Shared EPT tables"); #undef P - ret = scan_pci_devices(); - if ( ret ) - goto error; - ret = init_vtd_hw(); if ( ret ) goto error; diff --git a/xen/drivers/passthrough/x86/iommu.c b/xen/drivers/passthrough/x86/iommu.c index 0fa6dcc3fd..a7438c9c25 100644 --- a/xen/drivers/passthrough/x86/iommu.c +++ b/xen/drivers/passthrough/x86/iommu.c @@ -28,9 +28,15 @@ struct iommu_ops __read_mostly iommu_ops; int __init iommu_hardware_setup(void) { + int rc; + if ( !iommu_init_ops ) return -ENODEV; + rc = scan_pci_devices(); + if ( rc ) + return rc; + if ( !iommu_ops.init ) iommu_ops = *iommu_init_ops->ops; else From patchwork Tue Jul 16 10:16:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Durrant X-Patchwork-Id: 11045849 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 67B3A138D for ; Tue, 16 Jul 2019 10:19:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5731E28451 for ; Tue, 16 Jul 2019 10:19:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4B99F285A8; Tue, 16 Jul 2019 10:19:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A443128451 for ; Tue, 16 Jul 2019 10:19:12 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnKWA-0006HU-6g; Tue, 16 Jul 2019 10:17:06 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnKW8-0006HP-CB for xen-devel@lists.xenproject.org; Tue, 16 Jul 2019 10:17:04 +0000 X-Inumbo-ID: da54892c-a7b2-11e9-8980-bc764e045a96 Received: from esa5.hc3370-68.iphmx.com (unknown [216.71.155.168]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id da54892c-a7b2-11e9-8980-bc764e045a96; Tue, 16 Jul 2019 10:17:02 +0000 (UTC) Authentication-Results: esa5.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=paul.durrant@citrix.com; spf=Pass smtp.mailfrom=Paul.Durrant@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of paul.durrant@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="paul.durrant@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa5.hc3370-68.iphmx.com: domain of Paul.Durrant@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="Paul.Durrant@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa5.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa5.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: EM0GV+ecy1n5forbrenh4tHGmCp6dTs7nrOjO6/yuV0SxmJ3wViFOkygw18sOrlKD300j38BRf v/TjpPfMhPDPsBqkZUXb+IPmysM/X3rRXpr4YuGY5Tm8IVeF5qdUN7fINpbZtJMalCyUSDmYPO Qa/adRIwEa6Xu96nUJeBKN/1z3O3VYVE/EmYOBmGTUsAwYdGCW4CKFIw0IHmvGoaqxuaOtT7iC vkNWAVBdEIGrotg4Qjrn9aneM7aio5ngbsXNkVTFAOoGUnxDiihxjzk1qkHTNTTUkuDwaG/AZy PV0= X-SBRS: 2.7 X-MesageID: 3068124 X-Ironport-Server: esa5.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,498,1557201600"; d="scan'208";a="3068124" From: Paul Durrant To: Date: Tue, 16 Jul 2019 11:16:55 +0100 Message-ID: <20190716101657.23327-3-paul.durrant@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190716101657.23327-1-paul.durrant@citrix.com> References: <20190716101657.23327-1-paul.durrant@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 2/4] pci: add all-device iterator function... X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Stefano Stabellini , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , Tim Deegan , Julien Grall , Paul Durrant , Jan Beulich Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP ...and use it for setup_hwdom_pci_devices() and dump_pci_devices(). The unlock/process-pending-softirqs/lock sequence that was in _setup_hwdom_pci_devices() is now done in the generic iterator function, which does mean it is also done (unnecessarily) in the case of dump_pci_devices(), since run_all_nonirq_keyhandlers() will call process_pending_softirqs() before invoking each key handler anyway, but this is not performance critical code. The "==== segment XXXX ====" headline that was in _dump_pci_devices() has been dropped because it is non-trivial to deal with it when using a generic all-device iterator and, since the segment number is included in every log line anyway, it didn't add much value anyway. Signed-off-by: Paul Durrant Reviewed-by: "Roger Pau Monné" --- Cc: Jan Beulich Cc: Andrew Cooper Cc: George Dunlap Cc: Ian Jackson Cc: Julien Grall Cc: Konrad Rzeszutek Wilk Cc: Stefano Stabellini Cc: Tim Deegan Cc: Wei Liu v3: - Addressed review comments from Roger. v2: - New in v2. --- xen/drivers/passthrough/pci.c | 121 ++++++++++++++++++++++++------------------ xen/include/xen/pci.h | 1 + 2 files changed, 69 insertions(+), 53 deletions(-) diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index e88689425d..4bb9996049 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -1134,64 +1134,87 @@ static void __hwdom_init setup_one_hwdom_device(const struct setup_hwdom *ctxt, ctxt->d->domain_id, err); } -static int __hwdom_init _setup_hwdom_pci_devices(struct pci_seg *pseg, void *arg) +static int __hwdom_init setup_hwdom_pci_device(struct pci_dev *pdev, void *arg) { struct setup_hwdom *ctxt = arg; - int bus, devfn; + struct domain *d = ctxt->d; - for ( bus = 0; bus < 256; bus++ ) + if ( !pdev->domain ) + { + pdev->domain = d; + list_add(&pdev->domain_list, &d->pdev_list); + setup_one_hwdom_device(ctxt, pdev); + } + else if ( pdev->domain == dom_xen ) { - for ( devfn = 0; devfn < 256; devfn++ ) + pdev->domain = d; + setup_one_hwdom_device(ctxt, pdev); + pdev->domain = dom_xen; + } + else if ( pdev->domain != d ) + printk(XENLOG_WARNING "Dom%pd owning %04x:%02x:%02x.%u?\n", + pdev->domain, pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn)); + + return 0; +} + +struct psdi_ctxt { + int (*cb)(struct pci_dev *, void *); + void *arg; +}; + +static int pci_segment_devices_iterate(struct pci_seg *pseg, void *arg) +{ + struct psdi_ctxt *ctxt = arg; + unsigned int bus, devfn; + int rc = 0; + + /* + * We don't iterate by walking pseg->alldevs_list here because that + * would make the pcidevs_unlock()/lock() sequence below unsafe. + */ + for ( bus = 0; !rc && bus < 256; bus++ ) + for ( devfn = 0; !rc && devfn < 256; devfn++ ) { struct pci_dev *pdev = pci_get_pdev(pseg->nr, bus, devfn); if ( !pdev ) continue; - if ( !pdev->domain ) - { - pdev->domain = ctxt->d; - list_add(&pdev->domain_list, &ctxt->d->pdev_list); - setup_one_hwdom_device(ctxt, pdev); - } - else if ( pdev->domain == dom_xen ) - { - pdev->domain = ctxt->d; - setup_one_hwdom_device(ctxt, pdev); - pdev->domain = dom_xen; - } - else if ( pdev->domain != ctxt->d ) - printk(XENLOG_WARNING "Dom%d owning %04x:%02x:%02x.%u?\n", - pdev->domain->domain_id, pseg->nr, bus, - PCI_SLOT(devfn), PCI_FUNC(devfn)); - - if ( iommu_verbose ) - { - pcidevs_unlock(); - process_pending_softirqs(); - pcidevs_lock(); - } - } + rc = ctxt->cb(pdev, ctxt->arg); - if ( !iommu_verbose ) - { + /* + * Err on the safe side and assume the callback has taken + * a significant amount of time. + */ pcidevs_unlock(); process_pending_softirqs(); pcidevs_lock(); } - } - return 0; + return rc; +} + +int pci_pdevs_iterate(int (*cb)(struct pci_dev *, void *), void *arg) +{ + struct psdi_ctxt ctxt = { .cb = cb, .arg = arg }; + int rc; + + pcidevs_lock(); + rc = pci_segments_iterate(pci_segment_devices_iterate, &ctxt); + pcidevs_unlock(); + + return rc; } void __hwdom_init setup_hwdom_pci_devices( struct domain *d, int (*handler)(u8 devfn, struct pci_dev *)) { struct setup_hwdom ctxt = { .d = d, .handler = handler }; + int rc = pci_pdevs_iterate(setup_hwdom_pci_device, &ctxt); - pcidevs_lock(); - pci_segments_iterate(_setup_hwdom_pci_devices, &ctxt); - pcidevs_unlock(); + ASSERT(!rc); } #ifdef CONFIG_ACPI @@ -1294,24 +1317,18 @@ bool_t pcie_aer_get_firmware_first(const struct pci_dev *pdev) } #endif -static int _dump_pci_devices(struct pci_seg *pseg, void *arg) +static int dump_pci_device(struct pci_dev *pdev, void *arg) { - struct pci_dev *pdev; struct msi_desc *msi; - printk("==== segment %04x ====\n", pseg->nr); - - list_for_each_entry ( pdev, &pseg->alldevs_list, alldevs_list ) - { - printk("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < ", - pseg->nr, pdev->bus, - PCI_SLOT(pdev->devfn), PCI_FUNC(pdev->devfn), - pdev->domain ? pdev->domain->domain_id : -1, - (pdev->node != NUMA_NO_NODE) ? pdev->node : -1); - list_for_each_entry ( msi, &pdev->msi_list, list ) - printk("%d ", msi->irq); - printk(">\n"); - } + printk("%04x:%02x:%02x.%u - dom %-3d - node %-3d - MSIs < ", + pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn), + pdev->domain ? pdev->domain->domain_id : -1, + (pdev->node != NUMA_NO_NODE) ? pdev->node : -1); + list_for_each_entry ( msi, &pdev->msi_list, list ) + printk("%d ", msi->irq); + printk(">\n"); return 0; } @@ -1319,9 +1336,7 @@ static int _dump_pci_devices(struct pci_seg *pseg, void *arg) static void dump_pci_devices(unsigned char ch) { printk("==== PCI devices ====\n"); - pcidevs_lock(); - pci_segments_iterate(_dump_pci_devices, NULL); - pcidevs_unlock(); + pci_pdevs_iterate(dump_pci_device, NULL); } static int __init setup_dump_pcidevs(void) diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 04a9f46cc3..79eb25417b 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -154,6 +154,7 @@ int find_upstream_bridge(u16 seg, u8 *bus, u8 *devfn, u8 *secbus); struct pci_dev *pci_lock_pdev(int seg, int bus, int devfn); struct pci_dev *pci_lock_domain_pdev( struct domain *, int seg, int bus, int devfn); +int pci_pdevs_iterate(int (*cb)(struct pci_dev *, void *), void *arg); void setup_hwdom_pci_devices(struct domain *, int (*)(u8 devfn, struct pci_dev *)); From patchwork Tue Jul 16 10:16:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Paul Durrant X-Patchwork-Id: 11045847 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E59D813A4 for ; Tue, 16 Jul 2019 10:19:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D603A285A4 for ; Tue, 16 Jul 2019 10:19:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C8CC9285A9; Tue, 16 Jul 2019 10:19:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 39675285A8 for ; Tue, 16 Jul 2019 10:19:12 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnKWK-0006Iq-Ew; Tue, 16 Jul 2019 10:17:16 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnKWK-0006Ik-4E for xen-devel@lists.xenproject.org; Tue, 16 Jul 2019 10:17:16 +0000 X-Inumbo-ID: e01f46c4-a7b2-11e9-b8b3-674d6937a7d5 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id e01f46c4-a7b2-11e9-b8b3-674d6937a7d5; Tue, 16 Jul 2019 10:17:11 +0000 (UTC) Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=paul.durrant@citrix.com; spf=Pass smtp.mailfrom=Paul.Durrant@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa1.hc3370-68.iphmx.com: no sender authenticity information available from domain of paul.durrant@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa1.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="paul.durrant@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa1.hc3370-68.iphmx.com: domain of Paul.Durrant@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa1.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="Paul.Durrant@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa1.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa1.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: 9WZKrPeq98bI+0IMRRdMt8W/imAeMpt/QrvBoGwnBv2YlpmaRr6CbZKsjndp3UHBczpnyPobnH sOGVGAkMSMmQIGGOJIPDKpmIHpa+lNmylzlT1V0ikMXbdwxfSkaWKLwChFI4o5g5CpYPmvuW6E AhqqIkXnL0WHC6PQcGXRwkBazV8W8qYbOjAyGer44adtlkDIU7oxtxtAVUDqXdqtTOeuoMmeUV sw5qo/JnnUyKaW47PYbD/W3/xosaKw1jIY9pDjQpZfNtSnJ3PNAmJJvec9pBWiE+rK55Fmuqio pAs= X-SBRS: 2.7 X-MesageID: 3044883 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,498,1557201600"; d="scan'208";a="3044883" From: Paul Durrant To: Date: Tue, 16 Jul 2019 11:16:56 +0100 Message-ID: <20190716101657.23327-4-paul.durrant@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190716101657.23327-1-paul.durrant@citrix.com> References: <20190716101657.23327-1-paul.durrant@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 3/4] iommu: introduce iommu_groups X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Stefano Stabellini , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , Tim Deegan , Julien Grall , Paul Durrant , Jan Beulich Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Some devices may share a single PCIe initiator id, e.g. if they are actually legacy PCI devices behind a bridge, and hence DMA from such devices will be subject to the same address translation in the IOMMU. Hence these devices should be treated as a unit for the purposes of assignment. There are also other reasons why multiple devices should be treated as a unit, e.g. those subject to a shared RMRR or those downstream of a bridge that does not support ACS. This patch introduces a new struct iommu_group to act as a container for devices that should be treated as a unit, and builds a list of them as PCI devices are scanned. The iommu_ops already implement a method, get_device_group_id(), that is seemingly intended to return the initiator id for a given SBDF so use this as the mechanism for group assignment in the first instance. Assignment based on shared RMRR or lack of ACS will be dealt with in subsequent patches, as will modifications to the device assignment code. Signed-off-by: Paul Durrant Reviewed-by: "Roger Pau Monné" --- Cc: Jan Beulich Cc: Andrew Cooper Cc: George Dunlap Cc: Ian Jackson Cc: Julien Grall Cc: Konrad Rzeszutek Wilk Cc: Stefano Stabellini Cc: Tim Deegan Cc: Wei Liu v3: - Fix signed vs. unsigned issue spotted by Roger. v2: - Move code into new drivers/passthrough/groups.c - Drop the group index. - Handle failure to get group id. - Drop the group devs list. --- xen/drivers/passthrough/Makefile | 1 + xen/drivers/passthrough/groups.c | 91 +++++++++++++++++++++++++++++++++++++ xen/drivers/passthrough/x86/iommu.c | 8 +++- xen/include/xen/iommu.h | 7 +++ xen/include/xen/pci.h | 2 + 5 files changed, 108 insertions(+), 1 deletion(-) create mode 100644 xen/drivers/passthrough/groups.c diff --git a/xen/drivers/passthrough/Makefile b/xen/drivers/passthrough/Makefile index d50ab188c8..8a77110179 100644 --- a/xen/drivers/passthrough/Makefile +++ b/xen/drivers/passthrough/Makefile @@ -4,6 +4,7 @@ subdir-$(CONFIG_X86) += x86 subdir-$(CONFIG_ARM) += arm obj-y += iommu.o +obj-$(CONFIG_HAS_PCI) += groups.o obj-$(CONFIG_HAS_PCI) += pci.o obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o diff --git a/xen/drivers/passthrough/groups.c b/xen/drivers/passthrough/groups.c new file mode 100644 index 0000000000..c6d00980b6 --- /dev/null +++ b/xen/drivers/passthrough/groups.c @@ -0,0 +1,91 @@ +/* + * Copyright (c) 2019 Citrix Systems Inc. + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License as published by + * the Free Software Foundation; either version 2 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + */ + +#include +#include + +struct iommu_group { + unsigned int id; +}; + +static struct radix_tree_root iommu_groups; + +void __init iommu_groups_init(void) +{ + radix_tree_init(&iommu_groups); +} + +static struct iommu_group *alloc_iommu_group(unsigned int id) +{ + struct iommu_group *grp = xzalloc(struct iommu_group); + + if ( !grp ) + return NULL; + + grp->id = id; + + if ( radix_tree_insert(&iommu_groups, id, grp) ) + { + xfree(grp); + grp = NULL; + } + + return grp; +} + +static struct iommu_group *get_iommu_group(unsigned int id) +{ + struct iommu_group *grp = radix_tree_lookup(&iommu_groups, id); + + if ( !grp ) + grp = alloc_iommu_group(id); + + return grp; +} + +int iommu_group_assign(struct pci_dev *pdev, void *arg) +{ + const struct iommu_ops *ops = iommu_get_ops(); + int id; + struct iommu_group *grp; + + if ( !ops->get_device_group_id ) + return 0; + + id = ops->get_device_group_id(pdev->seg, pdev->bus, pdev->devfn); + if ( id < 0 ) + return -ENODATA; + + grp = get_iommu_group(id); + if ( !grp ) + return -ENOMEM; + + if ( iommu_verbose ) + printk(XENLOG_INFO "Assign %04x:%02x:%02x.%u -> IOMMU group %x\n", + pdev->seg, pdev->bus, PCI_SLOT(pdev->devfn), + PCI_FUNC(pdev->devfn), grp->id); + + pdev->grp = grp; + + return 0; +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/drivers/passthrough/x86/iommu.c b/xen/drivers/passthrough/x86/iommu.c index a7438c9c25..90fc750456 100644 --- a/xen/drivers/passthrough/x86/iommu.c +++ b/xen/drivers/passthrough/x86/iommu.c @@ -43,7 +43,13 @@ int __init iommu_hardware_setup(void) /* x2apic setup may have previously initialised the struct. */ ASSERT(iommu_ops.init == iommu_init_ops->ops->init); - return iommu_init_ops->setup(); + rc = iommu_init_ops->setup(); + if ( rc ) + return rc; + + iommu_groups_init(); + + return pci_pdevs_iterate(iommu_group_assign, NULL); } int iommu_enable_x2apic(void) diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h index 48f87480a7..c93f580fdc 100644 --- a/xen/include/xen/iommu.h +++ b/xen/include/xen/iommu.h @@ -317,6 +317,13 @@ DECLARE_PER_CPU(bool_t, iommu_dont_flush_iotlb); extern struct spinlock iommu_pt_cleanup_lock; extern struct page_list_head iommu_pt_cleanup_list; +#ifdef CONFIG_HAS_PCI + +void iommu_groups_init(void); +int iommu_group_assign(struct pci_dev *pdev, void *arg); + +#endif /* CONFIG_HAS_PCI */ + #endif /* _IOMMU_H_ */ /* diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h index 79eb25417b..e1f887af1c 100644 --- a/xen/include/xen/pci.h +++ b/xen/include/xen/pci.h @@ -79,6 +79,8 @@ struct pci_dev { struct list_head alldevs_list; struct list_head domain_list; + struct iommu_group *grp; + struct list_head msi_list; struct arch_msix *msix; From patchwork Tue Jul 16 10:16:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Paul Durrant X-Patchwork-Id: 11045843 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A68BD138D for ; Tue, 16 Jul 2019 10:18:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95AEF28451 for ; Tue, 16 Jul 2019 10:18:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 89A57285A4; Tue, 16 Jul 2019 10:18:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1460F28451 for ; Tue, 16 Jul 2019 10:18:51 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnKWI-0006IC-1P; Tue, 16 Jul 2019 10:17:14 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hnKWG-0006Hx-ST for xen-devel@lists.xenproject.org; Tue, 16 Jul 2019 10:17:12 +0000 X-Inumbo-ID: de3cd3f8-a7b2-11e9-b430-27c04cdaf338 Received: from esa1.hc3370-68.iphmx.com (unknown [216.71.145.142]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id de3cd3f8-a7b2-11e9-b430-27c04cdaf338; Tue, 16 Jul 2019 10:17:09 +0000 (UTC) Authentication-Results: esa1.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=paul.durrant@citrix.com; spf=Pass smtp.mailfrom=Paul.Durrant@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa1.hc3370-68.iphmx.com: no sender authenticity information available from domain of paul.durrant@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa1.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="paul.durrant@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa1.hc3370-68.iphmx.com: domain of Paul.Durrant@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa1.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="Paul.Durrant@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ~all" Received-SPF: None (esa1.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa1.hc3370-68.iphmx.com; envelope-from="Paul.Durrant@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: nKdheDnFVGQRsFfDpAtBta9To/MPyHceuQnoTo76CcDRT+J5hwjaBlV2AbeIulSMUbY+kMveTv giV6ZNU75DweLmfkWoaQdLv929rV9tTX1H2rzhQoD6pY3EK9o69VeEq77f1abawNMDAPOZ4qWJ M6lg7sQ36k8vvMdWjjFpa+zTgJUjdIcsPr06+/LzIvhtCm4RymFGwnd5FuiDe1wuRoNw9ePpHw /VwOONv/FIiqsI1Fh7EA9ztmNb56VcsjX5zvNV36ShI9ij/PVgHCRAiXtm8J086lkP4bGwJ2Ni /9E= X-SBRS: 2.7 X-MesageID: 3044880 X-Ironport-Server: esa1.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.63,498,1557201600"; d="scan'208";a="3044880" From: Paul Durrant To: Date: Tue, 16 Jul 2019 11:16:57 +0100 Message-ID: <20190716101657.23327-5-paul.durrant@citrix.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20190716101657.23327-1-paul.durrant@citrix.com> References: <20190716101657.23327-1-paul.durrant@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 4/4] iommu / pci: re-implement XEN_DOMCTL_get_device_group... X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Paul Durrant , Jan Beulich Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP ... using the new iommu_group infrastructure. Because 'sibling' devices are now members of the same iommu_group, implement the domctl by looking up the iommu_group of the pdev with the matching SBDF and then finding all the assigned pdevs that are in the group. Signed-off-by: Paul Durrant --- Cc: Jan Beulich v3: - Make 'max_sdevs' parameter in iommu_get_device_group() unsigned. - Add missing check of max_sdevs to avoid buffer overflow. v2: - Re-implement in the absence of a per-group devs list. - Make use of pci_sbdf_t. --- xen/drivers/passthrough/groups.c | 46 ++++++++++++++++++++++++++++++++++++ xen/drivers/passthrough/pci.c | 51 ++-------------------------------------- xen/include/xen/iommu.h | 3 +++ 3 files changed, 51 insertions(+), 49 deletions(-) diff --git a/xen/drivers/passthrough/groups.c b/xen/drivers/passthrough/groups.c index c6d00980b6..4e6e8022c1 100644 --- a/xen/drivers/passthrough/groups.c +++ b/xen/drivers/passthrough/groups.c @@ -12,8 +12,12 @@ * GNU General Public License for more details. */ +#include #include +#include #include +#include +#include struct iommu_group { unsigned int id; @@ -81,6 +85,48 @@ int iommu_group_assign(struct pci_dev *pdev, void *arg) return 0; } +int iommu_get_device_group(struct domain *d, pci_sbdf_t sbdf, + XEN_GUEST_HANDLE_64(uint32) buf, + unsigned int max_sdevs) +{ + struct iommu_group *grp = NULL; + struct pci_dev *pdev; + unsigned int i = 0; + + pcidevs_lock(); + + for_each_pdev ( d, pdev ) + { + if ( pdev->sbdf.sbdf == sbdf.sbdf ) + { + grp = pdev->grp; + break; + } + } + + if ( !grp ) + goto out; + + for_each_pdev ( d, pdev ) + { + if ( xsm_get_device_group(XSM_HOOK, pdev->sbdf.sbdf) || + pdev->grp != grp ) + continue; + + if ( i < max_sdevs && + unlikely(copy_to_guest_offset(buf, i++, &pdev->sbdf.sbdf, 1)) ) + { + pcidevs_unlock(); + return -EFAULT; + } + } + + out: + pcidevs_unlock(); + + return i; +} + /* * Local variables: * mode: C diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c index 4bb9996049..7171b50557 100644 --- a/xen/drivers/passthrough/pci.c +++ b/xen/drivers/passthrough/pci.c @@ -1564,53 +1564,6 @@ int deassign_device(struct domain *d, u16 seg, u8 bus, u8 devfn) return ret; } -static int iommu_get_device_group( - struct domain *d, u16 seg, u8 bus, u8 devfn, - XEN_GUEST_HANDLE_64(uint32) buf, int max_sdevs) -{ - const struct domain_iommu *hd = dom_iommu(d); - struct pci_dev *pdev; - int group_id, sdev_id; - u32 bdf; - int i = 0; - const struct iommu_ops *ops = hd->platform_ops; - - if ( !iommu_enabled || !ops || !ops->get_device_group_id ) - return 0; - - group_id = ops->get_device_group_id(seg, bus, devfn); - - pcidevs_lock(); - for_each_pdev( d, pdev ) - { - if ( (pdev->seg != seg) || - ((pdev->bus == bus) && (pdev->devfn == devfn)) ) - continue; - - if ( xsm_get_device_group(XSM_HOOK, (seg << 16) | (pdev->bus << 8) | pdev->devfn) ) - continue; - - sdev_id = ops->get_device_group_id(seg, pdev->bus, pdev->devfn); - if ( (sdev_id == group_id) && (i < max_sdevs) ) - { - bdf = 0; - bdf |= (pdev->bus & 0xff) << 16; - bdf |= (pdev->devfn & 0xff) << 8; - - if ( unlikely(copy_to_guest_offset(buf, i, &bdf, 1)) ) - { - pcidevs_unlock(); - return -1; - } - i++; - } - } - - pcidevs_unlock(); - - return i; -} - void iommu_dev_iotlb_flush_timeout(struct domain *d, struct pci_dev *pdev) { pcidevs_lock(); @@ -1667,11 +1620,11 @@ int iommu_do_pci_domctl( max_sdevs = domctl->u.get_device_group.max_sdevs; sdevs = domctl->u.get_device_group.sdev_array; - ret = iommu_get_device_group(d, seg, bus, devfn, sdevs, max_sdevs); + ret = iommu_get_device_group(d, PCI_SBDF3(seg, bus, devfn), sdevs, + max_sdevs); if ( ret < 0 ) { dprintk(XENLOG_ERR, "iommu_get_device_group() failed!\n"); - ret = -EFAULT; domctl->u.get_device_group.num_sdevs = 0; } else diff --git a/xen/include/xen/iommu.h b/xen/include/xen/iommu.h index c93f580fdc..6301833219 100644 --- a/xen/include/xen/iommu.h +++ b/xen/include/xen/iommu.h @@ -321,6 +321,9 @@ extern struct page_list_head iommu_pt_cleanup_list; void iommu_groups_init(void); int iommu_group_assign(struct pci_dev *pdev, void *arg); +int iommu_get_device_group(struct domain *d, pci_sbdf_t sbdf, + XEN_GUEST_HANDLE_64(uint32) buf, + unsigned int max_sdevs); #endif /* CONFIG_HAS_PCI */