From patchwork Wed Aug 23 04:56:10 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sinan Kaya X-Patchwork-Id: 9916579 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 386F360327 for ; Wed, 23 Aug 2017 04:57:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2915E28864 for ; Wed, 23 Aug 2017 04:57:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1C73328954; Wed, 23 Aug 2017 04:57:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_LOW autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9925A28864 for ; Wed, 23 Aug 2017 04:57:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=MP/JuMqA8lNie8Wu9cmGcpVlX745uNSLxXL4kBUZQJ4=; b=fsUgHzoubsaANQm0T+QWu2/9BK vePTzg9C4oAr5YDcDA4Xs87X6Op7qGFlWMPRjnlkpzOB9TYC7H463l6+NbsmcOE9DcHG9cubE9gvc /QPQl45LE/JYp6SZJoSMWkLJTulbwx2phGfrAwv/i7Q085EvWD+tg4Fr1hdn05o1EcUFFh4SKnOtM 3rdFGDvnPjcSJGGnFrd6hTKEko+ZamT2DjLi3DkExUyvPEwas6Y5ED/Uc6Av4tfUyqpoq7KYsv/QI Gm39aEVvVqfRBDVyIkWCH0SRcY5nV+Ox8d62RMNev5oHgi0at5dhPaxDYUWcJ15mrLscPfvMhst0r 1Y98Po7g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dkNjN-00036L-J9; Wed, 23 Aug 2017 04:57:29 +0000 Received: from smtp.codeaurora.org ([198.145.29.96]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1dkNiX-0002Hi-Em for linux-arm-kernel@lists.infradead.org; Wed, 23 Aug 2017 04:56:42 +0000 Received: by smtp.codeaurora.org (Postfix, from userid 1000) id D26666071F; Wed, 23 Aug 2017 04:56:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1503464180; bh=ZEIXopUjzQejHoXfFG8kVgovPgxiFGeFyHpsLZWMF9E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Bqw/VV8o/s2U1VwSN5yMQ6pzMpVrox8a0ZX3+8GGk3KeGotEfQ8rhIPmAhlzCBLOu hO/bvnUnh02ikgjl5W3OlPgOFcJvEwwsJupLCOG8z5WWhIDyngV/PHJbYpvY9vyrl6 KkEMSlXXkoFPsdGA3rJVl+jj+hYinAdknKU60ME0= Received: from drakthul.qualcomm.com (global_nat1_iad_fw.qualcomm.com [129.46.232.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: okaya@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id 4C18F60719; Wed, 23 Aug 2017 04:56:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1503464180; bh=ZEIXopUjzQejHoXfFG8kVgovPgxiFGeFyHpsLZWMF9E=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Bqw/VV8o/s2U1VwSN5yMQ6pzMpVrox8a0ZX3+8GGk3KeGotEfQ8rhIPmAhlzCBLOu hO/bvnUnh02ikgjl5W3OlPgOFcJvEwwsJupLCOG8z5WWhIDyngV/PHJbYpvY9vyrl6 KkEMSlXXkoFPsdGA3rJVl+jj+hYinAdknKU60ME0= DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 4C18F60719 Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=okaya@codeaurora.org From: Sinan Kaya To: linux-pci@vger.kernel.org, timur@codeaurora.org, alex.williamson@redhat.com Subject: [PATCH V12 4/5] PCI: Handle CRS ("device not ready") returned by device after FLR Date: Wed, 23 Aug 2017 00:56:10 -0400 Message-Id: <1503464171-6471-4-git-send-email-okaya@codeaurora.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1503464171-6471-1-git-send-email-okaya@codeaurora.org> References: <1503464171-6471-1-git-send-email-okaya@codeaurora.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170822_215637_631094_C5508743 X-CRM114-Status: GOOD ( 15.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sinan Kaya , linux-arm-msm@vger.kernel.org, Bjorn Helgaas , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Sporadic reset issues have been observed with Intel 750 NVMe drive while assigning the physical function to the guest machine. The sequence of events observed is as follows: - perform a Function Level Reset (FLR) - sleep up to 1000ms total - read ~0 from PCI_COMMAND - warn that the device didn't return from FLR - touch the device before it's ready - device drops config writes when we restore register settings - incomplete register restore leaves device in inconsistent state - device probe fails because device is in inconsistent state After reset, an endpoint may respond to config requests with Configuration Request Retry Status (CRS) to indicate that it is not ready to accept new requests. See PCIe r3.1, sec 2.3.1 and 6.6.2. After an FLR, read the Vendor ID and use pci_bus_wait_crs() to wait for a value that indicates the device is ready only if CRS visibility is supported and device is responding with 0x0001. If pci_bus_wait_crs() fails, i.e., the device has responded with CRS status for at least the timeout interval, fall back to the old behavior of reading the Command register until it is not ~0. Also increase the timeout value from 1 second to 60 seconds to have consistent behavior for root ports that do and do not support CRS visibility. Signed-off-by: Sinan Kaya --- drivers/pci/pci.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index af0cc34..5980ab3 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -3819,19 +3819,35 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev) */ static void pci_flr_wait(struct pci_dev *dev) { + int timeout_ms = 60000; int i = 0; u32 id; + /* + * Per PCIe r3.1, sec 6.6.2, the device should finish FLR within + * 100ms, but even after that, it may respond to config requests + * with CRS status if it requires more time. + */ + msleep(100); + + if (pci_bus_read_config_dword(dev->bus, dev->devfn, PCI_VENDOR_ID, &id)) + return; + + if (pci_bus_crs_visibility_pending(id) && + pci_bus_wait_crs(dev->bus, dev->devfn, id, timeout_ms)) + return; + + timeout_ms -= 100; do { msleep(100); pci_read_config_dword(dev, PCI_COMMAND, &id); - } while (i++ < 10 && id == ~0); + } while (i++ < (timeout_ms / 100) && id == ~0); if (id == ~0) dev_warn(&dev->dev, "Failed to return from FLR\n"); else if (i > 1) dev_info(&dev->dev, "Required additional %dms to return from FLR\n", - (i - 1) * 100); + i * 100); } /**