From patchwork Wed Feb 19 09:26:58 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 3679601 Return-Path: X-Original-To: patchwork-linux-arm@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id 5A441BF13A for ; Wed, 19 Feb 2014 09:27:41 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 5F1F720138 for ; Wed, 19 Feb 2014 09:27:40 +0000 (UTC) Received: from casper.infradead.org (casper.infradead.org [85.118.1.10]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 29F3420145 for ; Wed, 19 Feb 2014 09:27:39 +0000 (UTC) Received: from merlin.infradead.org ([2001:4978:20e::2]) by casper.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WG3RG-0003ub-NJ; Wed, 19 Feb 2014 09:27:34 +0000 Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1WG3RE-0000z1-7t; Wed, 19 Feb 2014 09:27:32 +0000 Received: from casper.infradead.org ([2001:770:15f::2]) by merlin.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1WG3RC-0000yZ-0R for linux-arm-kernel@merlin.infradead.org; Wed, 19 Feb 2014 09:27:30 +0000 Received: from top.free-electrons.com ([176.31.233.9] helo=mail.free-electrons.com) by casper.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1WG3RA-0003pk-89 for linux-arm-kernel@lists.infradead.org; Wed, 19 Feb 2014 09:27:28 +0000 Received: by mail.free-electrons.com (Postfix, from userid 106) id 6E0AA7AD; Wed, 19 Feb 2014 10:27:16 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Spam-Level: X-Spam-Status: No, score=-4.8 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=unavailable version=3.3.1 Received: from skate (col31-4-88-188-83-94.fbx.proxad.net [88.188.83.94]) by mail.free-electrons.com (Postfix) with ESMTPSA id C412A6E8; Wed, 19 Feb 2014 10:27:15 +0100 (CET) Date: Wed, 19 Feb 2014 10:26:58 +0100 From: Thomas Petazzoni To: Gerlando Falauto Subject: Re: pci-mvebu driver on km_kirkwood Message-ID: <20140219102658.76eec91e@skate> In-Reply-To: <53046D98.6020801@keymile.com> References: <51DD88A4.1030506@keymile.com> <20130710185706.72b124a4@skate> <51DD9A8C.10608@keymile.com> <20130711163220.2b3adf38@skate> <53039894.10905@keymile.com> <20140218212751.07c2aeb5@skate> <53046D98.6020801@keymile.com> Organization: Free Electrons X-Mailer: Claws Mail 3.9.1 (GTK+ 2.24.20; x86_64-pc-linux-gnu) Mime-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20140219_092728_312155_BAFC6AB7 X-CRM114-Status: GOOD ( 34.56 ) X-Spam-Score: -1.8 (-) Cc: Lior Amsalem , Andrew Lunn , Jason Cooper , "Longchamp, Valentin" , Ezequiel Garcia , "linux-arm-kernel@lists.infradead.org" , Sebastian Hesselbarth X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Dear Gerlando Falauto, On Wed, 19 Feb 2014 09:38:48 +0100, Gerlando Falauto wrote: > >> pci 0000:01:00.0: BAR 3: assigned [mem 0xe8000000-0xe87fffff] > >> pci 0000:01:00.0: BAR 4: assigned [mem 0xe8800000-0xe8801fff] > >> pci 0000:01:00.0: BAR 0: assigned [mem 0xe8802000-0xe8802fff] > >> pci 0000:01:00.0: BAR 2: assigned [mem 0xe8803000-0xe8803fff] > >> pci 0000:01:00.0: BAR 5: assigned [mem 0xe8804000-0xe8804fff] > > > > So in total, for the device 0000:01:00, the memory region should go > > from 0xe0000000 to 0xe8804fff. This means that a 256 MB window is > > needed for this device, because only power of two sizes are possible > > for MBus windows. > > > > Can you give me the output of /sys/kernel/debug/mvebu-mbus/devices ? It > > will tell us how the MBus windows are configured, as I suspect the > > problem might be here. > > Here it goes: > > [00] disabled > [01] disabled > [02] disabled > [03] disabled > [04] 00000000ff000000 - 00000000ff010000 : nand > [05] 00000000f4000000 - 00000000f8000000 : vpcie > [06] 00000000fe000000 - 00000000fe010000 : dragonite > [07] 00000000e0000000 - 00000000ec000000 : pcie0.0 > > So there's something wrong: a 256MB window should go all the way up to > 0xf0000000, and we have 192MB instead and I don't know how that would be > interpreted. My understanding is that a 192 MB window is illegal, because the window size should be encoded as a sequence of 1s followed by a sequence of 0s from the LSB to the MSB. To me, this means that only power of two window sizes are possible. > I couldn't figure out where this range comes from though, as in the > device tree I now have a size of 256MB (I stupidly set it to 192MB at > some point, but I now changed it): > > # hexdump -C /proc/device-tree/ocp@f1000000/pcie-controller/ranges > | cut -c1-58 > 00000000 82 00 00 00 00 00 00 00 00 04 00 00 00 04 00 00 > 00000010 00 00 00 00 00 00 20 00 82 00 00 00 00 00 00 00 > 00000020 e0 00 00 00 e0 00 00 00 00 00 00 00 10 00 00 00 > ^^^^^^^^^^^ Wow, that's an old DT representation that you have here :) But ok, let me try to explain. The 256 MB value that you define in the DT is the global PCIe memory aperture: it is the maximum amount of memory that we allow the PCIe driver to allocate for PCIe windows. But depending on which PCIe devices you have plugged in, and how large their BARs are, not necessarily all of these 256 MB will be used. So, you can very well have a 256 MB global PCIe memory aperture, and still have only one 1 MB PCIe memory window for PCIe 0.0 and a 256 KB PCIe memory window for PCIe 1.0, and that's it. Now, the 192 MB comes from the enumeration of your device. Linux enumerates the BAR of your device: pci 0000:01:00.0: BAR 1: assigned [mem 0xe0000000-0xe7ffffff] pci 0000:01:00.0: BAR 3: assigned [mem 0xe8000000-0xe87fffff] pci 0000:01:00.0: BAR 4: assigned [mem 0xe8800000-0xe8801fff] pci 0000:01:00.0: BAR 0: assigned [mem 0xe8802000-0xe8802fff] pci 0000:01:00.0: BAR 2: assigned [mem 0xe8803000-0xe8803fff] pci 0000:01:00.0: BAR 5: assigned [mem 0xe8804000-0xe8804fff] and then concludes that at the emulated bridge level, the memory region to be created is: pci 0000:00:01.0: bridge window [mem 0xe0000000-0xebffffff] which corresponds to the 192 MB window that we see created. But I believe a 192 MB memory window cannot work with MBus, it should be rounded up to the next power of 2. Can you try the below patch (not tested, not even compiled, might need some tweaks to apply to your 3.10 kernel) : I'm obviously interested in seeing the message that gets shown, as well as the new mvebu-mbus debugfs output. For good measure, if you could also dump the registers of the PCIe window. In your case, it was window 7, so dumping 0xf1020070 and 0xf1020074 would be useful. > But apart from that, what I still don't understand is how that could > have anything to do with my problem. The memory area I'm not able to > access starts at 0xe4000000. > BAR0, on the other hand, spawns 0xe8802000-0xe8802fff and seems to work > fine. I am not sure, but since we are configuring an invalid memory size, maybe the MBus behavior is undefined, and we get some completely funky behavior, where parts of the 192 MB window are actually work, but parts of it are not. Thomas diff --git a/drivers/pci/host/pci-mvebu.c b/drivers/pci/host/pci-mvebu.c index 13478ec..002229a 100644 --- a/drivers/pci/host/pci-mvebu.c +++ b/drivers/pci/host/pci-mvebu.c @@ -372,6 +372,11 @@ static void mvebu_pcie_handle_membase_change(struct mvebu_pcie_port *port) (((port->bridge.memlimit & 0xFFF0) << 16) | 0xFFFFF) - port->memwin_base; + pr_info("PCIE %d.%d: creating window at 0x%x, size 0x%x rounded up to 0x%x\n", + port->port, port->lane, port->memwin_base, + port->memwin_size, roundup_pow_of_two(port->memwin_size)); + port->memwin_size = roundup_pow_of_two(port->memwin_size); + mvebu_mbus_add_window_by_id(port->mem_target, port->mem_attr, port->memwin_base, port->memwin_size); }