From patchwork Thu May 23 05:29:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gibson X-Patchwork-Id: 10956901 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0398F112C for ; Thu, 23 May 2019 05:31:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D30B427FA8 for ; Thu, 23 May 2019 05:31:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C732727FC0; Thu, 23 May 2019 05:31:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 1E64727FA8 for ; Thu, 23 May 2019 05:31:44 +0000 (UTC) Received: from localhost ([127.0.0.1]:57928 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hTgKN-0001QH-Bw for patchwork-qemu-devel@patchwork.kernel.org; Thu, 23 May 2019 01:31:43 -0400 Received: from eggs.gnu.org ([209.51.188.92]:41329) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hTgIP-00080k-Bv for qemu-devel@nongnu.org; Thu, 23 May 2019 01:29:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hTgIM-0005jw-Mx for qemu-devel@nongnu.org; Thu, 23 May 2019 01:29:40 -0400 Received: from ozlabs.org ([2401:3900:2:1::2]:42073) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hTgIL-0005hz-2P; Thu, 23 May 2019 01:29:38 -0400 Received: by ozlabs.org (Postfix, from userid 1007) id 458dNM3BjTz9sBb; Thu, 23 May 2019 15:29:23 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1558589363; bh=6+yFFKzxCwb7KFLMf7OJjNXUm4RPq6kPWKgMltQOp3E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KdqarK2HGcHpAHgrWdxi2C6JbZ6pm0IF0nvCwVk4G0Z9jdozawx1Ot6eekLHMajwp 5RbE8WyFA42JyRwDziKnJFCCn1+tyoO8Rc7BhZTNKCM07F5HHBf0f15xWI+aS8y13r 5QWuH9SbfhQImxMDJ+6QxEYMXFSexWgvyhyTquUE= From: David Gibson To: qemu-ppc@nongnu.org Date: Thu, 23 May 2019 15:29:18 +1000 Message-Id: <20190523052918.1129-8-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190523052918.1129-1-david@gibson.dropbear.id.au> References: <20190523052918.1129-1-david@gibson.dropbear.id.au> MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2401:3900:2:1::2 Subject: [Qemu-devel] [PATCH 8/8] spapr: Allow hot plug/unplug of PCI bridges and devices under PCI bridges X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mst@redhat.com, qemu-devel@nongnu.org, groug@kaod.org, clg@kaod.org, mdroth@linux.ibm.com, David Gibson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP The pseries machine type already allows PCI hotplug and unplug via the PAPR mechanism, but only on the root bus of each PHB. This patch extends this to allow PCI to PCI bridges to be hotplugged, and devices to be hotplugged or unplugged under P2P bridges. For now we disallow hot unplugging P2P bridges. I tried doing that, but haven't managed to get it working, I think due to some guest side problems that need further investigation. To do this we dynamically construct DRCs when bridges are hot (or cold) added, which can in turn be used to hotplug devices under the bridge. Signed-off-by: David Gibson --- hw/ppc/spapr_pci.c | 115 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 102 insertions(+), 13 deletions(-) diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c index 94691fcfc2..a4d5e46525 100644 --- a/hw/ppc/spapr_pci.c +++ b/hw/ppc/spapr_pci.c @@ -1257,30 +1257,53 @@ static SpaprDrc *drc_from_dev(SpaprPhbState *phb, PCIDevice *dev) return drc_from_devfn(phb, chassis, dev->devfn); } -static void add_drcs(SpaprPhbState *phb) +static void add_drcs(SpaprPhbState *phb, PCIBus *bus, Error **errp) { + Object *owner; int i; + uint8_t chassis; + Error *local_err = NULL; if (!phb->dr_enabled) { return; } + chassis = chassis_from_bus(bus, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + + if (pci_bus_is_root(bus)) { + owner = OBJECT(phb); + } else { + owner = OBJECT(pci_bridge_get_device(bus)); + } + for (i = 0; i < PCI_SLOT_MAX * PCI_FUNC_MAX; i++) { - spapr_dr_connector_new(OBJECT(phb), TYPE_SPAPR_DRC_PCI, - drc_id_from_devfn(phb, 0, i)); + spapr_dr_connector_new(owner, TYPE_SPAPR_DRC_PCI, + drc_id_from_devfn(phb, chassis, i)); } } -static void remove_drcs(SpaprPhbState *phb) +static void remove_drcs(SpaprPhbState *phb, PCIBus *bus, Error **errp) { int i; + uint8_t chassis; + Error *local_err = NULL; if (!phb->dr_enabled) { return; } + chassis = chassis_from_bus(bus, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + for (i = PCI_SLOT_MAX * PCI_FUNC_MAX - 1; i >= 0; i--) { - SpaprDrc *drc = drc_from_devfn(phb, 0, i); + SpaprDrc *drc = drc_from_devfn(phb, chassis, i); if (drc) { object_unparent(OBJECT(drc)); @@ -1325,6 +1348,7 @@ static int spapr_dt_pci_bus(SpaprPhbState *sphb, PCIBus *bus, .sphb = sphb, .err = 0, }; + int ret; _FDT(fdt_setprop_cell(fdt, offset, "#address-cells", RESOURCE_CELLS_ADDRESS)); @@ -1339,6 +1363,12 @@ static int spapr_dt_pci_bus(SpaprPhbState *sphb, PCIBus *bus, } } + ret = spapr_dt_drc(fdt, offset, OBJECT(bus->parent_dev), + SPAPR_DR_CONNECTOR_TYPE_PCI); + if (ret) { + return ret; + } + return offset; } @@ -1483,11 +1513,26 @@ int spapr_pci_dt_populate(SpaprDrc *drc, SpaprMachineState *spapr, return 0; } +static void spapr_pci_bridge_plug(SpaprPhbState *phb, + PCIBridge *bridge, + Error **errp) +{ + Error *local_err = NULL; + PCIBus *bus = pci_bridge_get_sec_bus(bridge); + + add_drcs(phb, bus, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } +} + static void spapr_pci_plug(HotplugHandler *plug_handler, DeviceState *plugged_dev, Error **errp) { SpaprPhbState *phb = SPAPR_PCI_HOST_BRIDGE(DEVICE(plug_handler)); PCIDevice *pdev = PCI_DEVICE(plugged_dev); + PCIDeviceClass *pc = PCI_DEVICE_GET_CLASS(plugged_dev); SpaprDrc *drc = drc_from_dev(phb, pdev); Error *local_err = NULL; PCIBus *bus = PCI_BUS(qdev_get_parent_bus(DEVICE(pdev))); @@ -1509,6 +1554,14 @@ static void spapr_pci_plug(HotplugHandler *plug_handler, g_assert(drc); + if (pc->is_bridge) { + spapr_pci_bridge_plug(phb, PCI_BRIDGE(plugged_dev), &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } + } + /* Following the QEMU convention used for PCIe multifunction * hotplug, we do not allow functions to be hotplugged to a * slot that already has function 0 present @@ -1559,9 +1612,26 @@ out: error_propagate(errp, local_err); } +static void spapr_pci_bridge_unplug(SpaprPhbState *phb, + PCIBridge *bridge, + Error **errp) +{ + Error *local_err = NULL; + PCIBus *bus = pci_bridge_get_sec_bus(bridge); + + remove_drcs(phb, bus, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } +} + static void spapr_pci_unplug(HotplugHandler *plug_handler, DeviceState *plugged_dev, Error **errp) { + PCIDeviceClass *pc = PCI_DEVICE_GET_CLASS(plugged_dev); + SpaprPhbState *phb = SPAPR_PCI_HOST_BRIDGE(DEVICE(plug_handler)); + /* some version guests do not wait for completion of a device * cleanup (generally done asynchronously by the kernel) before * signaling to QEMU that the device is safe, but instead sleep @@ -1573,6 +1643,16 @@ static void spapr_pci_unplug(HotplugHandler *plug_handler, * an 'idle' state, as the device cleanup code expects. */ pci_device_reset(PCI_DEVICE(plugged_dev)); + + if (pc->is_bridge) { + Error *local_err = NULL; + spapr_pci_bridge_unplug(phb, PCI_BRIDGE(plugged_dev), &local_err); + if (local_err) { + error_propagate(errp, local_err); + } + return; + } + object_property_set_bool(OBJECT(plugged_dev), false, "realized", NULL); } @@ -1593,6 +1673,7 @@ static void spapr_pci_unplug_request(HotplugHandler *plug_handler, g_assert(drc->dev == plugged_dev); if (!spapr_drc_unplug_requested(drc)) { + PCIDeviceClass *pc = PCI_DEVICE_GET_CLASS(plugged_dev); uint32_t slotnr = PCI_SLOT(pdev->devfn); SpaprDrc *func_drc; SpaprDrcClass *func_drck; @@ -1606,6 +1687,10 @@ static void spapr_pci_unplug_request(HotplugHandler *plug_handler, return; } + if (pc->is_bridge) { + error_setg(errp, "PCI: Hot unplug of PCI bridges not supported"); + } + /* ensure any other present functions are pending unplug */ if (PCI_FUNC(pdev->devfn) == 0) { for (i = 1; i < 8; i++) { @@ -1658,6 +1743,7 @@ static void spapr_phb_unrealize(DeviceState *dev, Error **errp) SpaprTceTable *tcet; int i; const unsigned windows_supported = spapr_phb_windows_supported(sphb); + Error *local_err = NULL; spapr_phb_nvgpu_free(sphb); @@ -1678,7 +1764,11 @@ static void spapr_phb_unrealize(DeviceState *dev, Error **errp) } } - remove_drcs(sphb); + remove_drcs(sphb, phb->bus, &local_err); + if (local_err) { + error_propagate(errp, local_err); + return; + } for (i = PCI_NUM_PINS - 1; i >= 0; i--) { if (sphb->lsi_table[i].irq) { @@ -1743,6 +1833,7 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp) uint64_t msi_window_size = 4096; SpaprTceTable *tcet; const unsigned windows_supported = spapr_phb_windows_supported(sphb); + Error *local_err = NULL; if (!spapr) { error_setg(errp, TYPE_SPAPR_PCI_HOST_BRIDGE " needs a pseries machine"); @@ -1879,7 +1970,6 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp) /* Initialize the LSI table */ for (i = 0; i < PCI_NUM_PINS; i++) { uint32_t irq = SPAPR_IRQ_PCI_LSI + sphb->index * PCI_NUM_PINS + i; - Error *local_err = NULL; if (smc->legacy_irq_allocation) { irq = spapr_irq_findone(spapr, &local_err); @@ -1904,7 +1994,11 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp) } /* allocate connectors for child PCI devices */ - add_drcs(sphb); + add_drcs(sphb, phb->bus, &local_err); + if (local_err) { + error_propagate(errp, local_err); + goto unrealize; + } /* DMA setup */ for (i = 0; i < windows_supported; ++i) { @@ -2320,11 +2414,6 @@ int spapr_dt_phb(SpaprPhbState *phb, uint32_t intc_phandle, void *fdt, return ret; } - ret = spapr_dt_drc(fdt, bus_off, OBJECT(phb), SPAPR_DR_CONNECTOR_TYPE_PCI); - if (ret) { - return ret; - } - spapr_phb_nvgpu_populate_dt(phb, fdt, bus_off, &errp); if (errp) { error_report_err(errp);