From patchwork Thu Oct 6 03:03:04 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Gibson X-Patchwork-Id: 9363631 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D733D600C8 for ; Thu, 6 Oct 2016 03:06:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BCD5A28717 for ; Thu, 6 Oct 2016 03:06:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AC6DC28DB5; Thu, 6 Oct 2016 03:06:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6435C28717 for ; Thu, 6 Oct 2016 03:06:39 +0000 (UTC) Received: from localhost ([::1]:52677 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1brz14-0002Pw-II for patchwork-qemu-devel@patchwork.kernel.org; Wed, 05 Oct 2016 23:06:38 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45352) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bryxs-0000KB-A1 for qemu-devel@nongnu.org; Wed, 05 Oct 2016 23:03:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bryxo-00031o-Us for qemu-devel@nongnu.org; Wed, 05 Oct 2016 23:03:19 -0400 Received: from ozlabs.org ([103.22.144.67]:46667) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bryxo-00030C-D4; Wed, 05 Oct 2016 23:03:16 -0400 Received: by ozlabs.org (Postfix, from userid 1007) id 3sqHXK4ynxz9svs; Thu, 6 Oct 2016 14:03:13 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gibson.dropbear.id.au; s=201602; t=1475722993; bh=oFl+kBP+YeJzGHS00qiXUUgERbre5e/Effk0ISMRkA0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ke9ImAw+itKspq5KPo7Y8Kmy0IlZLr/y8QQHkLviCT7QD4aaHXVg/d1vpDrqmfeEf SifYyXusDP4uY0aH3u/L30F8Fjub5pPFsepMoWtplgxauGTUFYrpTuDB7zyJdWV/86 pqDpqRrTLSjsj/F+EmhzeQrX+/sx9g7L4Z4wJ1BI= From: David Gibson To: qemu-ppc@nongnu.org Date: Thu, 6 Oct 2016 14:03:04 +1100 Message-Id: <1475722987-18644-2-git-send-email-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1475722987-18644-1-git-send-email-david@gibson.dropbear.id.au> References: <1475722987-18644-1-git-send-email-david@gibson.dropbear.id.au> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 103.22.144.67 Subject: [Qemu-devel] [RFC 1/4] spapr_pci: Delegate placement of PCI host bridges to machine type X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lvivier@redhat.com, thuth@redhat.com, mdroth@linux.vnet.ibm.com, nikunj@linux.vnet.ibm.com, mst@redhat.com, aik@ozlabs.ru, qemu-devel@nongnu.org, agraf@suse.de, abologna@redhat.com, bharata@linux.vnet.ibm.com, mpolednik@redhat.com, David Gibson Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP The 'spapr-pci-host-bridge' represents the virtual PCI host bridge (PHB) for a PAPR guest. Unlike on x86, it's routine on Power (both bare metal and PAPR guests) to have numerous independent PHBs, each controlling a separate PCI domain. There are two ways of configuring the spapr-pci-host-bridge device: first it can be done fully manually, specifying the locations and sizes of all the IO windows. This gives the most control, but is very awkward with 6 mandatory parameters. Alternatively just an "index" can be specified which essentially selects from an array of predefined PHB locations. The PHB at index 0 is automatically created as the default PHB. The current set of default locations causes some problems for guests with large RAM (> 1 TiB) or PCI devices with very large BARs (e.g. big nVidia GPGPU cards via VFIO). Obviously, for migration we can only change the locations on a new machine type, however. This is awkward, because the placement is currently decided within the spapr-pci-host-bridge code, so it breaks abstraction to look inside the machine type version. So, this patch delegates the "default mode" PHB placement from the spapr-pci-host-bridge device back to the machine type via a public method in sPAPRMachineClass. It's still a bit ugly, but it's about the best we can do. For now, this just changes where the calculation is done. It doesn't change the actual location of the host bridges, or any other behaviour. Signed-off-by: David Gibson --- hw/ppc/spapr.c | 34 ++++++++++++++++++++++++++++++++++ hw/ppc/spapr_pci.c | 22 ++++++++-------------- include/hw/pci-host/spapr.h | 11 +---------- include/hw/ppc/spapr.h | 4 ++++ 4 files changed, 47 insertions(+), 24 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index 03e3803..f6e9c2a 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -2370,6 +2370,39 @@ static HotpluggableCPUList *spapr_query_hotpluggable_cpus(MachineState *machine) return head; } +static void spapr_phb_placement(sPAPRMachineState *spapr, uint32_t index, + uint64_t *buid, hwaddr *pio, hwaddr *pio_size, + hwaddr *mmio, hwaddr *mmio_size, + unsigned n_dma, uint32_t *liobns, Error **errp) +{ + const uint64_t base_buid = 0x800000020000000ULL; + const hwaddr phb0_base = 0x10000000000ULL; /* 1 TiB */ + const hwaddr phb_spacing = 0x1000000000ULL; /* 64 GiB */ + const hwaddr mmio_offset = 0xa0000000; /* 2 GiB + 512 MiB */ + const hwaddr pio_offset = 0x80000000; /* 2 GiB */ + const uint32_t max_index = 255; + + hwaddr phb_base; + int i; + + if (index > max_index) { + error_setg(errp, "\"index\" for PAPR PHB is too large (max %u)", + max_index); + return; + } + + *buid = base_buid + index; + for (i = 0; i < n_dma; ++i) { + liobns[i] = SPAPR_PCI_LIOBN(index, i); + } + + phb_base = phb0_base + index * phb_spacing; + *pio = phb_base + pio_offset; + *pio_size = SPAPR_PCI_IO_WIN_SIZE; + *mmio = phb_base + mmio_offset; + *mmio_size = SPAPR_PCI_MMIO_WIN_SIZE; +} + static void spapr_machine_class_init(ObjectClass *oc, void *data) { MachineClass *mc = MACHINE_CLASS(oc); @@ -2406,6 +2439,7 @@ static void spapr_machine_class_init(ObjectClass *oc, void *data) mc->query_hotpluggable_cpus = spapr_query_hotpluggable_cpus; fwc->get_dev_path = spapr_get_fw_dev_path; nc->nmi_monitor_handler = spapr_nmi; + smc->phb_placement = spapr_phb_placement; } static const TypeInfo spapr_machine_info = { diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c index 4f00865..c0fc964 100644 --- a/hw/ppc/spapr_pci.c +++ b/hw/ppc/spapr_pci.c @@ -1311,7 +1311,8 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp) sphb->ddw_enabled ? SPAPR_PCI_DMA_MAX_WINDOWS : 1; if (sphb->index != (uint32_t)-1) { - hwaddr windows_base; + sPAPRMachineClass *smc = SPAPR_MACHINE_GET_CLASS(spapr); + Error *local_err = NULL; if ((sphb->buid != (uint64_t)-1) || (sphb->dma_liobn[0] != (uint32_t)-1) || (sphb->dma_liobn[1] != (uint32_t)-1 && windows_supported == 2) @@ -1322,21 +1323,14 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp) return; } - if (sphb->index > SPAPR_PCI_MAX_INDEX) { - error_setg(errp, "\"index\" for PAPR PHB is too large (max %u)", - SPAPR_PCI_MAX_INDEX); + smc->phb_placement(spapr, sphb->index, + &sphb->buid, &sphb->io_win_addr, &sphb->io_win_size, + &sphb->mem_win_addr, &sphb->mem_win_size, + windows_supported, sphb->dma_liobn, &local_err); + if (local_err) { + error_propagate(errp, local_err); return; } - - sphb->buid = SPAPR_PCI_BASE_BUID + sphb->index; - for (i = 0; i < windows_supported; ++i) { - sphb->dma_liobn[i] = SPAPR_PCI_LIOBN(sphb->index, i); - } - - windows_base = SPAPR_PCI_WINDOW_BASE - + sphb->index * SPAPR_PCI_WINDOW_SPACING; - sphb->mem_win_addr = windows_base + SPAPR_PCI_MMIO_WIN_OFF; - sphb->io_win_addr = windows_base + SPAPR_PCI_IO_WIN_OFF; } if (sphb->buid == (uint64_t)-1) { diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h index 30dbd46..8c9ebfd 100644 --- a/include/hw/pci-host/spapr.h +++ b/include/hw/pci-host/spapr.h @@ -79,18 +79,9 @@ struct sPAPRPHBState { uint32_t numa_node; }; -#define SPAPR_PCI_MAX_INDEX 255 - -#define SPAPR_PCI_BASE_BUID 0x800000020000000ULL - #define SPAPR_PCI_MEM_WIN_BUS_OFFSET 0x80000000ULL -#define SPAPR_PCI_WINDOW_BASE 0x10000000000ULL -#define SPAPR_PCI_WINDOW_SPACING 0x1000000000ULL -#define SPAPR_PCI_MMIO_WIN_OFF 0xA0000000 -#define SPAPR_PCI_MMIO_WIN_SIZE (SPAPR_PCI_WINDOW_SPACING - \ - SPAPR_PCI_MEM_WIN_BUS_OFFSET) -#define SPAPR_PCI_IO_WIN_OFF 0x80000000 +#define SPAPR_PCI_MMIO_WIN_SIZE 0xf80000000 #define SPAPR_PCI_IO_WIN_SIZE 0x10000 #define SPAPR_PCI_MSI_WINDOW 0x40000000000ULL diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index 39dadaa..9e1c5a5 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -40,6 +40,10 @@ struct sPAPRMachineClass { bool dr_lmb_enabled; /* enable dynamic-reconfig/hotplug of LMBs */ bool use_ohci_by_default; /* use USB-OHCI instead of XHCI */ const char *tcg_default_cpu; /* which (TCG) CPU to simulate by default */ + void (*phb_placement)(sPAPRMachineState *spapr, uint32_t index, + uint64_t *buid, hwaddr *pio, hwaddr *pio_size, + hwaddr *mmio, hwaddr *mmio_size, + unsigned n_dma, uint32_t *liobns, Error **errp); }; /**