From patchwork Fri Aug 18 21:32:17 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bjorn Helgaas X-Patchwork-Id: 9910001 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E9955600CC for ; Fri, 18 Aug 2017 21:33:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB1AF28C18 for ; Fri, 18 Aug 2017 21:33:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D00CE28D6E; Fri, 18 Aug 2017 21:33:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_LOW autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4483828C18 for ; Fri, 18 Aug 2017 21:33:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:To:From:Subject:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=3yIK11+HrywRmAl/HDakLV3fBed8SwMVYx5lguN+IZA=; b=MGVuq2r2aAixb5 hTIbvWCWggcm62UjUTxcqr5IYzfEInsnPHwj0S7ff/JuH3sp9PCSP2EIrHuAJYcbx4P1Y5O9rzrPX nO0U8X96eeYXm0HGm8nPVADZOHZMmoFkb9igwJITunCUu79wZIJ0JUALuDbrwG5ecCXtv5VqXqxy/ IlTHNhOdSv3ehEgRQqn8seJ0mG7cSbESkM4kZPdl5Yl4H08m1FaCjxIURB1ZMiB9CwgvZllFkGRvg aXWnZJWt5cZXtr/xvAq/q3Ab1V+nv2wdSkBaighlbwAydiGBDGWTIYfBRTdgG7ZbimK4TzUcVA30s YwfOOY9LxiLI/Bth8+tg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1diotb-0008Pf-VC; Fri, 18 Aug 2017 21:33:36 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1diosg-0007C0-V1 for linux-arm-kernel@lists.infradead.org; Fri, 18 Aug 2017 21:32:42 +0000 Received: from localhost (173-24-227-107.client.mchsi.com [173.24.227.107]) (using TLSv1.2 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id BAAD8218F9; Fri, 18 Aug 2017 21:32:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BAAD8218F9 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=helgaas@kernel.org Subject: [PATCH v11 3/4] PCI: Handle CRS ("device not ready") returned by device after FLR From: Bjorn Helgaas To: Sinan Kaya Date: Fri, 18 Aug 2017 16:32:17 -0500 Message-ID: <20170818213217.15145.38764.stgit@bhelgaas-glaptop.roam.corp.google.com> In-Reply-To: <20170818212310.15145.21732.stgit@bhelgaas-glaptop.roam.corp.google.com> References: <20170818212310.15145.21732.stgit@bhelgaas-glaptop.roam.corp.google.com> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170818_143239_341362_F5C0C975 X-CRM114-Status: GOOD ( 16.18 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-pci@vger.kernel.org, Timur Tabi , linux-kernel@vger.kernel.org, Alex Williamson , linux-arm-msm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Sinan Kaya XXX I think this needs to somehow use the same timeout for the PCI_COMMAND loop as for the CRS part, so this works on machines without CRS Software Visibility. -- bhelgaas Sporadic reset issues have been observed with Intel 750 NVMe drive while assigning the physical function to the guest machine. The sequence of events observed is as follows: - perform a Function Level Reset (FLR) - sleep up to 1000ms total - read ~0 from PCI_COMMAND - warn that the device didn't return from FLR - touch the device before it's ready - device drops config writes when we restore register settings - incomplete register restore leaves device in inconsistent state - device probe fails because device is in inconsistent state After reset, an endpoint may respond to config requests with Configuration Request Retry Status (CRS) to indicate that it is not ready to accept new requests. See PCIe r3.1, sec 2.3.1 and 6.6.2. After an FLR, read the Vendor ID and use pci_bus_wait_crs() to wait for a value that indicates the device is ready. If pci_bus_wait_crs() fails, i.e., the device has responded with CRS status for at least the timeout interval, fall back to the old behavior of reading the Command register until it is not ~0. Signed-off-by: Sinan Kaya [bhelgaas: changelog, adjust for Vendor ID test being inside pci_bus_wait_crs(), drop PCI_COMMAND tweaks] Not-Signed-off-by: Bjorn Helgaas --- drivers/pci/pci.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index af0cc3456dc1..34c0aa1f37aa 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -3821,17 +3821,32 @@ static void pci_flr_wait(struct pci_dev *dev) { int i = 0; u32 id; + bool ret; + + /* + * Per PCIe r3.1, sec 6.6.2, the device should finish FLR within + * 100ms, but even after that, it may respond to config requests + * with CRS status if it requires more time. + */ + msleep(100); + + if (pci_bus_read_config_dword(dev->bus, dev->devfn, PCI_VENDOR_ID, &id)) + return; + + ret = pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); + if (ret) + return; do { msleep(100); pci_read_config_dword(dev, PCI_COMMAND, &id); - } while (i++ < 10 && id == ~0); + } while (i++ < 9 && id == ~0); if (id == ~0) dev_warn(&dev->dev, "Failed to return from FLR\n"); else if (i > 1) dev_info(&dev->dev, "Required additional %dms to return from FLR\n", - (i - 1) * 100); + i * 100); } /**