diff mbox

Intel I350 mini-PCIe card (igb) on Mirabox (mvebu / Armada 370)

Message ID alpine.DEB.2.10.1403280056250.4521@vroombuntu (mailing list archive)
State New, archived
Headers show

Commit Message

Neil Greatorex March 28, 2014, 1:03 a.m. UTC 
Jason, Thomas,

With the help you've given, I think I've identified what the problem is. 
It is basically what Thomas suggested the problem was (that he'd seen with 
the Armada 38x). The call to mvebu_pcie_set_local_dev_nr causes the PCIe 
link to reset. In my tests with this Intel card, it takes ~25ms for the 
link to go down, and then another ~13ms for it to come back up.

I have put together a patch below that I would be interested to get your 
opinions on and also for you to test against any cards you may have. I 
only have the one mini PCIe card so it's not a great test!

With my card, it is now detected correctly, and there are no issues when 
rescanning the PCI bus.

Cheers,
Neil


From a0ca1552e737c18b5651fe1f27ade76f512ca191 Mon Sep 17 00:00:00 2001
From: Neil Greatorex <neil@fatboyfat.co.uk>
Date: Fri, 28 Mar 2014 00:35:04 +0000
Subject: [PATCH] PCI: mvebu: Wait for PCIe link to reset after setting
  device number

After setting the local device nr, the PCIe link may go down briefly. If 
we don't wait for the link to come back up we may miss that device when we 
scan the bus (assuming it is not present).

This patch waits up to 40ms for the link to go down, and if it did go down,
waits up to 100ms for it to come back up. If the link was down to begin with,
it will not wait at all.

Signed-off-by: Neil Greatorex <neil@fatboyfat.co.uk>
---
  drivers/pci/host/pci-mvebu.c | 57 
++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 57 insertions(+)

  			continue;
@@ -975,8 +976,64 @@ static int mvebu_pcie_probe(struct platform_device 
*pdev)
  			continue;
  		}

+		/* After setting the local dev number, the PCI-E link may go
+		 * down for a period of time. If we don't wait for it to come
+		 * back up, then we risk missing enumerating that device when
+		 * we scan the bus. However, if the link was down to begin with,
+		 * there's no point waiting, so check now.
+		 */
+		link_present = mvebu_pcie_link_up(port);
+
  		mvebu_pcie_set_local_dev_nr(port, 1);

+		if (link_present) {
+			/* In testing, the link took ~25ms to go down, and
+			 * another ~15ms to come back up, so wait ~40ms for the
+			 * link to go down, and then up to ~100ms for it to come
+			 * back up. If the link doesn't go down after ~40ms, then
+			 * it probably won't go down at all, so carry on.
+			 */
+			if (mvebu_pcie_link_up(port)) {
+				while (retry_count < 40) {
+					if (!mvebu_pcie_link_up(port))
+						break;
+					retry_count++;
+					mdelay(1);
+				}
+
+				if (retry_count == 40) {
+					dev_dbg(&pdev->dev,
+						"PCIe%d.%d: after setting dev nr, link stayed up\n",
+						port->port, port->lane);
+				} else {
+					dev_info(&pdev->dev,
+						 "PCIe%d.%d: after setting dev nr, link went down after %d polls\n",
+						 port->port, port->lane, retry_count);
+				}
+			}
+
+			retry_count = 0;
+
+			if (!mvebu_pcie_link_up(port)) {
+				while (retry_count < 100) {
+					if (mvebu_pcie_link_up(port))
+						break;
+					retry_count++;
+					mdelay(1);
+				}
+
+				if (retry_count == 100) {
+					dev_info(&pdev->dev,
+						"PCIe%d.%d: after setting dev nr, link failed to come back up (timeout)\n",
+						port->port, port->lane);
+				} else {
+					dev_info(&pdev->dev,
+						 "PCIe%d.%d: after setting dev nr, link came back up after %d polls ",
+						 port->port, port->lane, retry_count);
+				}
+			}
+		}
+
  		port->dn = child;
  		spin_lock_init(&port->conf_lock);
  		mvebu_sw_pci_bridge_init(port);

Comments

Jason Gunthorpe March 28, 2014, 2:04 a.m. UTC | #1 
On Fri, Mar 28, 2014 at 01:03:55AM +0000, Neil Greatorex wrote:

> With the help you've given, I think I've identified what the problem
> is. It is basically what Thomas suggested the problem was (that he'd
> seen with the Armada 38x). The call to mvebu_pcie_set_local_dev_nr
> causes the PCIe link to reset. In my tests with this Intel card, it
> takes ~25ms for the link to go down, and then another ~13ms for it
> to come back up.

Wow, fascinating - not only does the link go down, but it takes 25ms,
and it seemed based on the patch I sent you, that the first config op
triggers this reset immediately..

If I have time I'll look in the spec, maybe the devnumber is encoded
in low level flow control packets or something that makes sense of
this..

What does your kernel boot log look like? Does the USB PEX link behave
the same?? Presumably it happens faster?

A different option would be to force the link down, then change the
dev number, then bring it back up. At least that way the process is
under the control of the driver and we are not waiting for the
mysterious 25 ms. Or maybe change the dev number, then trigger a
link reset...

> I have put together a patch below that I would be interested to get
> your opinions on and also for you to test against any cards you may
> have. I only have the one mini PCIe card so it's not a great test!

Unfortunately my systems here never boot with the PEX up so I cannot
easily verify..

One suggestion about the patch I have is to move everything you added
into the mvebu_pcie_set_local_dev_nr function and bail if the
devnumber is already correct. That way the bootloader could set it
properly and avoid this.

Regards,
Jason
diff mbox

Patch

diff --git a/drivers/pci/host/pci-mvebu.c b/drivers/pci/host/pci-mvebu.c
index 0e79665..afd0dce 100644
--- a/drivers/pci/host/pci-mvebu.c
+++ b/drivers/pci/host/pci-mvebu.c
@@ -891,6 +891,7 @@  static int mvebu_pcie_probe(struct platform_device 
*pdev)
  	for_each_child_of_node(pdev->dev.of_node, child) {
  		struct mvebu_pcie_port *port = &pcie->ports[i];
  		enum of_gpio_flags flags;
+		u32 link_present, retry_count = 0;

  		if (!of_device_is_available(child))