From patchwork Thu Mar 3 01:42:53 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Kardashevskiy X-Patchwork-Id: 8487191 Return-Path: X-Original-To: patchwork-qemu-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 5F3EF9F372 for ; Thu, 3 Mar 2016 01:44:22 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 7B4E220351 for ; Thu, 3 Mar 2016 01:44:21 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 38DB6201C8 for ; Thu, 3 Mar 2016 01:44:19 +0000 (UTC) Received: from localhost ([::1]:59993 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1abIJN-00046N-Ij for patchwork-qemu-devel@patchwork.kernel.org; Wed, 02 Mar 2016 20:44:17 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45573) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1abIJD-00045l-Vb for qemu-devel@nongnu.org; Wed, 02 Mar 2016 20:44:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1abIJ9-0002YB-Q3 for qemu-devel@nongnu.org; Wed, 02 Mar 2016 20:44:07 -0500 Received: from e23smtp08.au.ibm.com ([202.81.31.141]:49800) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1abIJ8-0002Xl-VZ for qemu-devel@nongnu.org; Wed, 02 Mar 2016 20:44:03 -0500 Received: from localhost by e23smtp08.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 3 Mar 2016 11:43:59 +1000 Received: from d23dlp01.au.ibm.com (202.81.31.203) by e23smtp08.au.ibm.com (202.81.31.205) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 3 Mar 2016 11:43:55 +1000 X-IBM-Helo: d23dlp01.au.ibm.com X-IBM-MailFrom: aik@ozlabs.ru X-IBM-RcptTo: qemu-devel@nongnu.org;qemu-ppc@nongnu.org Received: from d23relay07.au.ibm.com (d23relay07.au.ibm.com [9.190.26.37]) by d23dlp01.au.ibm.com (Postfix) with ESMTP id 97C2E2CE8059; Thu, 3 Mar 2016 12:43:54 +1100 (EST) Received: from d23av02.au.ibm.com (d23av02.au.ibm.com [9.190.235.138]) by d23relay07.au.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u231hi8e62193894; Thu, 3 Mar 2016 12:43:54 +1100 Received: from d23av02.au.ibm.com (localhost [127.0.0.1]) by d23av02.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u231hK37022341; Thu, 3 Mar 2016 12:43:20 +1100 Received: from ozlabs.au.ibm.com (ozlabs.au.ibm.com [9.192.253.14]) by d23av02.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id u231hJV4021541; Thu, 3 Mar 2016 12:43:19 +1100 Received: from bran.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) by ozlabs.au.ibm.com (Postfix) with ESMTP id A3328A0225; Thu, 3 Mar 2016 12:42:55 +1100 (AEDT) Received: from vpl2.ozlabs.ibm.com (vpl2.ozlabs.ibm.com [10.61.141.27]) by bran.ozlabs.ibm.com (Postfix) with ESMTP id 8FB35E39EA; Thu, 3 Mar 2016 12:42:55 +1100 (AEDT) From: Alexey Kardashevskiy To: qemu-devel@nongnu.org Date: Thu, 3 Mar 2016 12:42:53 +1100 Message-Id: <1456969373-6741-1-git-send-email-aik@ozlabs.ru> X-Mailer: git-send-email 2.5.0.rc3 X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16030301-0029-0000-0000-00004427194B X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 202.81.31.141 Cc: Alexey Kardashevskiy , qemu-ppc@nongnu.org, Alexander Graf , David Gibson Subject: [Qemu-devel] [PATCH qemu] spapr-pci: Make MMIO spacing a machine property and increase it X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The pseries machine supports multiple PHBs. Each PHB's MMIO/IO space is mapped to the CPU address space starting at SPAPR_PCI_WINDOW_BASE plus some offset which is calculated from PHB's index and SPAPR_PCI_WINDOW_SPACING which is defined now as 64GB. Since the default 32bit DMA window is using first 2GB of MMIO space, the amount of MMIO which the PCI devices can actually use is reduced to 62GB. This is a problem if the user wants to use devices with huge BARs. For example, 2 PCI functions of a NVIDIA K80 adapter being passed through will exceed this limit as they have 16M + 16G + 32M BARs which (when aligned) will need 64GB. This converts SPAPR_PCI_WINDOW_BASE and SPAPR_PCI_WINDOW_SPACING to sPAPRMachineState properties. This uses old values for pseries machine before 2.6 and increases the spacing to 128GB so MMIO space becomes 126GB. This changes the default value of sPAPRPHBState::mem_win_size to -1 for pseries-2.6 and adds setup to spapr_phb_realize. Signed-off-by: Alexey Kardashevskiy --- hw/ppc/spapr.c | 43 ++++++++++++++++++++++++++++++++++++++++++- hw/ppc/spapr_pci.c | 14 ++++++++++---- include/hw/pci-host/spapr.h | 4 +--- include/hw/ppc/spapr.h | 1 + 4 files changed, 54 insertions(+), 8 deletions(-) diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c index e9d4abf..d21ad8a 100644 --- a/hw/ppc/spapr.c +++ b/hw/ppc/spapr.c @@ -40,6 +40,7 @@ #include "migration/migration.h" #include "mmu-hash64.h" #include "qom/cpu.h" +#include "qapi/visitor.h" #include "hw/boards.h" #include "hw/ppc/ppc.h" @@ -2100,6 +2101,29 @@ static void spapr_set_kvm_type(Object *obj, const char *value, Error **errp) spapr->kvm_type = g_strdup(value); } +static void spapr_prop_get_uint64(Object *obj, Visitor *v, const char *name, + void *opaque, Error **errp) +{ + uint64_t value = *(uint64_t *)opaque; + visit_type_uint64(v, name, &value, errp); +} + +static void spapr_prop_set_uint64(Object *obj, Visitor *v, const char *name, + void *opaque, Error **errp) +{ + uint64_t value = -1; + visit_type_uint64(v, name, &value, errp); + *(uint64_t *)opaque = value; +} + +static void spapr_prop_add_uint64(Object *obj, const char *name, + uint64_t *pval, const char *desc) +{ + object_property_add(obj, name, "uint64", spapr_prop_get_uint64, + spapr_prop_set_uint64, NULL, pval, NULL); + object_property_set_description(obj, name, desc, NULL); +} + static void spapr_machine_initfn(Object *obj) { sPAPRMachineState *spapr = SPAPR_MACHINE(obj); @@ -2110,6 +2134,10 @@ static void spapr_machine_initfn(Object *obj) object_property_set_description(obj, "kvm-type", "Specifies the KVM virtualization mode (HV, PR)", NULL); + spapr_prop_add_uint64(obj, "phb-mmio-base", &spapr->phb_mmio_base, + "Base address for PCI host bridge MMIO"); + spapr_prop_add_uint64(obj, "phb-mmio-spacing", &spapr->phb_mmio_spacing, + "Amount of MMIO space per PCI host bridge"); } static void spapr_machine_finalizefn(Object *obj) @@ -2357,6 +2385,10 @@ static const TypeInfo spapr_machine_info = { */ static void spapr_machine_2_6_instance_options(MachineState *machine) { + sPAPRMachineState *spapr = SPAPR_MACHINE(machine); + + spapr->phb_mmio_base = SPAPR_PCI_WINDOW_BASE; + spapr->phb_mmio_spacing = SPAPR_PCI_WINDOW_SPACING; } static void spapr_machine_2_6_class_options(MachineClass *mc) @@ -2370,10 +2402,19 @@ DEFINE_SPAPR_MACHINE(2_6, "2.6", true); * pseries-2.5 */ #define SPAPR_COMPAT_2_5 \ - HW_COMPAT_2_5 + HW_COMPAT_2_5 \ + {\ + .driver = TYPE_SPAPR_PCI_HOST_BRIDGE,\ + .property = "mem_win_size",\ + .value = "0x1000000000",\ + }, static void spapr_machine_2_5_instance_options(MachineState *machine) { + sPAPRMachineState *spapr = SPAPR_MACHINE(machine); + + spapr->phb_mmio_base = 0x10000000000ULL; + spapr->phb_mmio_spacing = 0x1000000000ULL; } static void spapr_machine_2_5_class_options(MachineClass *mc) diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c index e8edad3..bae01dd 100644 --- a/hw/ppc/spapr_pci.c +++ b/hw/ppc/spapr_pci.c @@ -1260,9 +1260,11 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp) sphb->buid = SPAPR_PCI_BASE_BUID + sphb->index; sphb->dma_liobn = SPAPR_PCI_LIOBN(sphb->index, 0); - windows_base = SPAPR_PCI_WINDOW_BASE - + sphb->index * SPAPR_PCI_WINDOW_SPACING; + windows_base = spapr->phb_mmio_base + + sphb->index * spapr->phb_mmio_spacing; sphb->mem_win_addr = windows_base + SPAPR_PCI_MMIO_WIN_OFF; + sphb->mem_win_size = spapr->phb_mmio_spacing - + SPAPR_PCI_MEM_WIN_BUS_OFFSET; sphb->io_win_addr = windows_base + SPAPR_PCI_IO_WIN_OFF; } @@ -1281,6 +1283,11 @@ static void spapr_phb_realize(DeviceState *dev, Error **errp) return; } + if (sphb->mem_win_size == (hwaddr)-1) { + error_setg(errp, "Memory window size not specified for PHB"); + return; + } + if (sphb->io_win_addr == (hwaddr)-1) { error_setg(errp, "IO window address not specified for PHB"); return; @@ -1441,8 +1448,7 @@ static Property spapr_phb_properties[] = { DEFINE_PROP_UINT64("buid", sPAPRPHBState, buid, -1), DEFINE_PROP_UINT32("liobn", sPAPRPHBState, dma_liobn, -1), DEFINE_PROP_UINT64("mem_win_addr", sPAPRPHBState, mem_win_addr, -1), - DEFINE_PROP_UINT64("mem_win_size", sPAPRPHBState, mem_win_size, - SPAPR_PCI_MMIO_WIN_SIZE), + DEFINE_PROP_UINT64("mem_win_size", sPAPRPHBState, mem_win_size, -1), DEFINE_PROP_UINT64("io_win_addr", sPAPRPHBState, io_win_addr, -1), DEFINE_PROP_UINT64("io_win_size", sPAPRPHBState, io_win_size, SPAPR_PCI_IO_WIN_SIZE), diff --git a/include/hw/pci-host/spapr.h b/include/hw/pci-host/spapr.h index 7de5e02..b828c31 100644 --- a/include/hw/pci-host/spapr.h +++ b/include/hw/pci-host/spapr.h @@ -107,10 +107,8 @@ struct sPAPRPHBVFIOState { #define SPAPR_PCI_MEM_WIN_BUS_OFFSET 0x80000000ULL #define SPAPR_PCI_WINDOW_BASE 0x10000000000ULL -#define SPAPR_PCI_WINDOW_SPACING 0x1000000000ULL +#define SPAPR_PCI_WINDOW_SPACING 0x2000000000ULL #define SPAPR_PCI_MMIO_WIN_OFF 0xA0000000 -#define SPAPR_PCI_MMIO_WIN_SIZE (SPAPR_PCI_WINDOW_SPACING - \ - SPAPR_PCI_MEM_WIN_BUS_OFFSET) #define SPAPR_PCI_IO_WIN_OFF 0x80000000 #define SPAPR_PCI_IO_WIN_SIZE 0x10000 diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h index 098d85d..8b1369e 100644 --- a/include/hw/ppc/spapr.h +++ b/include/hw/ppc/spapr.h @@ -48,6 +48,7 @@ struct sPAPRMachineState { struct VIOsPAPRBus *vio_bus; QLIST_HEAD(, sPAPRPHBState) phbs; + uint64_t phb_mmio_base, phb_mmio_spacing; struct sPAPRNVRAM *nvram; XICSState *icp; DeviceState *rtc;