From patchwork Fri Mar 7 20:08:44 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yinghai Lu X-Patchwork-Id: 3794491 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: X-Original-To: patchwork-linux-pci@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 70ECDBF540 for ; Fri, 7 Mar 2014 20:07:26 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 4D420202EC for ; Fri, 7 Mar 2014 20:07:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C48E3201F0 for ; Fri, 7 Mar 2014 20:07:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751197AbaCGUHW (ORCPT ); Fri, 7 Mar 2014 15:07:22 -0500 Received: from aserp1040.oracle.com ([141.146.126.69]:28263 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751011AbaCGUHW (ORCPT ); Fri, 7 Mar 2014 15:07:22 -0500 Received: from acsinet22.oracle.com (acsinet22.oracle.com [141.146.126.238]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id s27K76V1002530 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 7 Mar 2014 20:07:07 GMT Received: from userz7021.oracle.com (userz7021.oracle.com [156.151.31.85]) by acsinet22.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id s27K73QR015329 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL); Fri, 7 Mar 2014 20:07:04 GMT Received: from abhmp0010.oracle.com (abhmp0010.oracle.com [141.146.116.16]) by userz7021.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id s27K72s7002709; Fri, 7 Mar 2014 20:07:02 GMT Received: from linux-siqj.site (/10.132.126.191) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 07 Mar 2014 12:07:02 -0800 From: Yinghai Lu To: Bjorn Helgaas Cc: Benjamin Herrenschmidt , Wei Yang , Gavin Shan , Jack Morgenstein , amirv@mellanox.com, Or Gerlitz , eugenia@mellanox.com, Guo Chao , talal@mellanox.com, linux-pci@vger.kernel.org, Yinghai Lu Subject: [PATCH v7] PCI: Try best to allocate pref mmio 64bit above 4g Date: Fri, 7 Mar 2014 12:08:44 -0800 Message-Id: <1394222924-28886-1-git-send-email-yinghai@kernel.org> X-Mailer: git-send-email 1.8.4.5 In-Reply-To: X-Source-IP: acsinet22.oracle.com [141.146.126.238] Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When one of children resources does not support MEM_64, MEM_64 for bridge get reset, so pull down whole pref resource on the bridge under 4G. If the bridge support pref mem 64, will only allocate that with pref mem64 to children that support it. For children resources if they only support pref mem 32, will allocate them from non pref mem instead. If the bridge only support 32bit pref mmio, will still have all children pref mmio under that. -v2: Add release bridge res support with bridge mem res for pref_mem children res. -v3: refresh and make it can be applied early before for_each_dev_res patchset. -v4: fix non-pref mmio 64bit support found by Guo Chao. -v7: repost as ibm's powerpc system work again when 1. petiboot boot kernel have this patch. 2. or mellanox driver added new .shutdown member. discussion could be found at: http://marc.info/?t=138968064500001&r=1&w=2 Signed-off-by: Yinghai Lu Tested-by: Guo Chao Tested-by: Wei Yang --- drivers/pci/setup-bus.c | 138 ++++++++++++++++++++++++++++++++---------------- drivers/pci/setup-res.c | 20 ++++++ 2 files changed, 111 insertions(+), 47 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Index: linux-2.6/drivers/pci/setup-bus.c =================================================================== --- linux-2.6.orig/drivers/pci/setup-bus.c +++ linux-2.6/drivers/pci/setup-bus.c @@ -713,12 +713,11 @@ static void pci_bridge_check_ranges(stru bus resource of a given type. Note: we intentionally skip the bus resources which have already been assigned (that is, have non-NULL parent resource). */ -static struct resource *find_free_bus_resource(struct pci_bus *bus, unsigned long type) +static struct resource *find_free_bus_resource(struct pci_bus *bus, + unsigned long type_mask, unsigned long type) { int i; struct resource *r; - unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM | - IORESOURCE_PREFETCH; pci_bus_for_each_resource(bus, r, i) { if (r == &ioport_resource || r == &iomem_resource) @@ -815,7 +814,8 @@ static void pbus_size_io(struct pci_bus resource_size_t add_size, struct list_head *realloc_head) { struct pci_dev *dev; - struct resource *b_res = find_free_bus_resource(bus, IORESOURCE_IO); + struct resource *b_res = find_free_bus_resource(bus, IORESOURCE_IO, + IORESOURCE_IO); resource_size_t size = 0, size0 = 0, size1 = 0; resource_size_t children_add_size = 0; resource_size_t min_align, align; @@ -915,15 +915,17 @@ static inline resource_size_t calculate_ * guarantees that all child resources fit in this size. */ static int pbus_size_mem(struct pci_bus *bus, unsigned long mask, - unsigned long type, resource_size_t min_size, - resource_size_t add_size, - struct list_head *realloc_head) + unsigned long type, unsigned long type2, + unsigned long type3, + resource_size_t min_size, resource_size_t add_size, + struct list_head *realloc_head) { struct pci_dev *dev; resource_size_t min_align, align, size, size0, size1; resource_size_t aligns[12]; /* Alignments from 1Mb to 2Gb */ int order, max_order; - struct resource *b_res = find_free_bus_resource(bus, type); + struct resource *b_res = find_free_bus_resource(bus, + mask | IORESOURCE_PREFETCH, type); unsigned int mem64_mask = 0; resource_size_t children_add_size = 0; @@ -944,7 +946,9 @@ static int pbus_size_mem(struct pci_bus struct resource *r = &dev->resource[i]; resource_size_t r_size; - if (r->parent || (r->flags & mask) != type) + if (r->parent || ((r->flags & mask) != type && + (r->flags & mask) != type2 && + (r->flags & mask) != type3)) continue; r_size = resource_size(r); #ifdef CONFIG_PCI_IOV @@ -1117,8 +1121,9 @@ void __ref __pci_bus_size_bridges(struct struct list_head *realloc_head) { struct pci_dev *dev; - unsigned long mask, prefmask; + unsigned long mask, prefmask, type2 = 0, type3 = 0; resource_size_t additional_mem_size = 0, additional_io_size = 0; + struct resource *b_res; list_for_each_entry(dev, &bus->devices, bus_list) { struct pci_bus *b = dev->subordinate; @@ -1163,15 +1168,34 @@ void __ref __pci_bus_size_bridges(struct has already been allocated by arch code, try non-prefetchable range for both types of PCI memory resources. */ + b_res = &bus->self->resource[PCI_BRIDGE_RESOURCES]; mask = IORESOURCE_MEM; prefmask = IORESOURCE_MEM | IORESOURCE_PREFETCH; - if (pbus_size_mem(bus, prefmask, prefmask, + if (b_res[2].flags & IORESOURCE_MEM_64) { + prefmask |= IORESOURCE_MEM_64; + if (pbus_size_mem(bus, prefmask, prefmask, + prefmask, prefmask, realloc_head ? 0 : additional_mem_size, - additional_mem_size, realloc_head)) - mask = prefmask; /* Success, size non-prefetch only. */ - else - additional_mem_size += additional_mem_size; - pbus_size_mem(bus, mask, IORESOURCE_MEM, + additional_mem_size, realloc_head)) { + /* Success, size non-pref64 only. */ + mask = prefmask; + type2 = prefmask & ~IORESOURCE_MEM_64; + type3 = prefmask & ~IORESOURCE_PREFETCH; + } + } + if (!type2) { + prefmask &= ~IORESOURCE_MEM_64; + if (pbus_size_mem(bus, prefmask, prefmask, + prefmask, prefmask, + realloc_head ? 0 : additional_mem_size, + additional_mem_size, realloc_head)) { + /* Success, size non-prefetch only. */ + mask = prefmask; + } else + additional_mem_size += additional_mem_size; + type2 = type3 = IORESOURCE_MEM; + } + pbus_size_mem(bus, mask, IORESOURCE_MEM, type2, type3, realloc_head ? 0 : additional_mem_size, additional_mem_size, realloc_head); break; @@ -1257,42 +1281,66 @@ static void __ref __pci_bridge_assign_re static void pci_bridge_release_resources(struct pci_bus *bus, unsigned long type) { - int idx; - bool changed = false; - struct pci_dev *dev; + struct pci_dev *dev = bus->self; struct resource *r; unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM | - IORESOURCE_PREFETCH; + IORESOURCE_PREFETCH | IORESOURCE_MEM_64; + unsigned old_flags = 0; + struct resource *b_res; + int idx = 1; - dev = bus->self; - for (idx = PCI_BRIDGE_RESOURCES; idx <= PCI_BRIDGE_RESOURCE_END; - idx++) { - r = &dev->resource[idx]; - if ((r->flags & type_mask) != type) - continue; - if (!r->parent) - continue; - /* - * if there are children under that, we should release them - * all - */ - release_child_resources(r); - if (!release_resource(r)) { - dev_printk(KERN_DEBUG, &dev->dev, - "resource %d %pR released\n", idx, r); - /* keep the old size */ - r->end = resource_size(r) - 1; - r->start = 0; - r->flags = 0; - changed = true; - } - } + b_res = &dev->resource[PCI_BRIDGE_RESOURCES]; + + /* + * 1. if there is io port assign fail, will release bridge + * io port. + * 2. if there is non pref mmio assign fail, release bridge + * nonpref mmio. + * 3. if there is 64bit pref mmio assign fail, and bridge pref + * is 64bit, release bridge pref mmio. + * 4. if there is pref mmio assign fail, and bridge pref is + * 32bit mmio, release bridge pref mmio + * 5. if there is pref mmio assign fail, and bridge pref is not + * assigned, release bridge nonpref mmio. + */ + if (type & IORESOURCE_IO) + idx = 0; + else if (!(type & IORESOURCE_PREFETCH)) + idx = 1; + else if ((type & IORESOURCE_MEM_64) && + (b_res[2].flags & IORESOURCE_MEM_64)) + idx = 2; + else if (!(b_res[2].flags & IORESOURCE_MEM_64) && + (b_res[2].flags & IORESOURCE_PREFETCH)) + idx = 2; + else + idx = 1; + + r = &b_res[idx]; + + if (!r->parent) + return; + + /* + * if there are children under that, we should release them + * all + */ + release_child_resources(r); + if (!release_resource(r)) { + type = old_flags = r->flags & type_mask; + dev_printk(KERN_DEBUG, &dev->dev, "resource %d %pR released\n", + PCI_BRIDGE_RESOURCES + idx, r); + /* keep the old size */ + r->end = resource_size(r) - 1; + r->start = 0; + r->flags = 0; - if (changed) { /* avoiding touch the one without PREF */ if (type & IORESOURCE_PREFETCH) type = IORESOURCE_PREFETCH; __pci_setup_bridge(bus, type); + /* for next child res under same bridge */ + r->flags = old_flags; } } @@ -1471,7 +1519,7 @@ void pci_assign_unassigned_root_bus_reso LIST_HEAD(fail_head); struct pci_dev_resource *fail_res; unsigned long type_mask = IORESOURCE_IO | IORESOURCE_MEM | - IORESOURCE_PREFETCH; + IORESOURCE_PREFETCH | IORESOURCE_MEM_64; int pci_try_num = 1; enum enable_type enable_local; Index: linux-2.6/drivers/pci/setup-res.c =================================================================== --- linux-2.6.orig/drivers/pci/setup-res.c +++ linux-2.6/drivers/pci/setup-res.c @@ -208,15 +208,31 @@ static int __pci_assign_resource(struct /* First, try exact prefetching match.. */ ret = pci_bus_alloc_resource(bus, res, size, align, min, - IORESOURCE_PREFETCH, + IORESOURCE_PREFETCH | IORESOURCE_MEM_64, pcibios_align_resource, dev); - if (ret < 0 && (res->flags & IORESOURCE_PREFETCH)) { + if (ret < 0 && + (res->flags & (IORESOURCE_PREFETCH | IORESOURCE_MEM_64)) == + (IORESOURCE_PREFETCH | IORESOURCE_MEM_64)) { + /* + * That failed. + * + * Try below 4g pref + */ + ret = pci_bus_alloc_resource(bus, res, size, align, min, + IORESOURCE_PREFETCH, + pcibios_align_resource, dev); + } + + if (ret < 0 && + (res->flags & (IORESOURCE_PREFETCH | IORESOURCE_MEM_64))) { /* * That failed. * * But a prefetching area can handle a non-prefetching * window (it will just not perform as well). + * + * Also can put 64bit under 32bit range. (below 4g). */ ret = pci_bus_alloc_resource(bus, res, size, align, min, 0, pcibios_align_resource, dev);