From patchwork Fri Aug 11 16:56:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sinan Kaya X-Patchwork-Id: 9896371 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id AF00860236 for ; Fri, 11 Aug 2017 17:04:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9BA2527D29 for ; Fri, 11 Aug 2017 17:04:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 902A928B42; Fri, 11 Aug 2017 17:04:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_LOW autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E51A527D29 for ; Fri, 11 Aug 2017 17:04:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=OTY7ekTV2yZOe13HR2wqJ8nJzUk6Mj+mhSuTxjqcAd8=; b=WsbWyjAheX/6EljpALvVo64uye RnCwt7HEETLt1X13C3nh6eZdHPyJosUVi7D4KnftpvBgqcVorKANKwYj5PrD38ROD0S/rRoabJ0Qe DmgYYqrs70KS7Cpqdwxbx4ArirovL4UtYlRlBNdWqvg958HQv8v7BYeD3NHtCyCQgp8kaw4QKoDOp c4zelu69vrpGsgbDWdzLizvso6q6urpPmcMlB/tYL+qfDa7xfWxB8E3wXyHtCd9FVrKh8k7bd4UWd 44g/Cu7MBug6SqHI0NeF/vpft3yJcfTn9vfyxuXavsw+cPhOeSA2Jfw65avNHWwfszvqsUaDpgBh+ WLNux2nQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1dgDLm-0005n2-Or; Fri, 11 Aug 2017 17:03:54 +0000 Received: from smtp.codeaurora.org ([198.145.29.96]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1dgDFA-00030T-0g for linux-arm-kernel@lists.infradead.org; Fri, 11 Aug 2017 16:57:07 +0000 Received: by smtp.codeaurora.org (Postfix, from userid 1000) id 73365601A1; Fri, 11 Aug 2017 16:56:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1502470605; bh=Z1JxWXLoCBVi5wTJgty9W4F5nCGu/7DgRGEl/B3n4Kc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=maLjNCYDIXBqG197uk3S7HID8INsVSACKe/e6ghdgZiAy5/pVWjcdV18HG0rfpUL7 hVjJPP5xBfmI30lxggV04WR81GZNj1SvasGCoHmCL7FR3si4PjzjLN7N/e76wiaanT gGuQ6q5lzNNpPZK9B33R4a6nO8r4J6uNqGbSxzUc= Received: from drakthul.qualcomm.com (global_nat1_iad_fw.qualcomm.com [129.46.232.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: okaya@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id E9408601A1; Fri, 11 Aug 2017 16:56:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1502470604; bh=Z1JxWXLoCBVi5wTJgty9W4F5nCGu/7DgRGEl/B3n4Kc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gA2wDd4cBIb9Mm2DUAq/oRyFqB6KKpCoH/c6zbG9Hb/KJ70VeGPXyIwhF8/3/vb4Q XuprTCcK8s9pd0fYIy32Pf4jsCr/qzOAYNaZU5qiiMAe+8lqvUnd139gS6PmrwN8JF alSXT+dRBL95KziVvDqmBhZV6XD37TN3el8roIGU= DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org E9408601A1 Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=okaya@codeaurora.org From: Sinan Kaya To: linux-pci@vger.kernel.org, timur@codeaurora.org, alex.williamson@redhat.com Subject: [PATCH V10 2/3] PCI: handle CRS returned by device after FLR Date: Fri, 11 Aug 2017 12:56:35 -0400 Message-Id: <1502470596-4112-2-git-send-email-okaya@codeaurora.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1502470596-4112-1-git-send-email-okaya@codeaurora.org> References: <1502470596-4112-1-git-send-email-okaya@codeaurora.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170811_095704_224670_673092D8 X-CRM114-Status: GOOD ( 16.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sinan Kaya , linux-arm-msm@vger.kernel.org, Bjorn Helgaas , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Sporadic reset issues have been observed with Intel 750 NVMe drive while assigning the physical function to the guest machine. The sequence of events observed is as follows: - perform a Function Level Reset (FLR) - sleep up to 1000ms total - read ~0 from PCI_COMMAND - warn that the device didn't return from FLR - touch the device before it's ready - drop register read and writes performing register settings restore - incomplete reset operation and partial register restoration - second time device probe fails in the guest machine as HW is left in limbo. An endpoint is allowed to issue Configuration Request Retry Status (CRS) following a FLR request to indicate that it is not ready to accept new requests. CRS is defined in PCIe r3.1, sec 2.3.1. Request Handling Rules and CRS usage in FLR context is mentioned in PCIe r3.1, sec 6.6.2. Function-Level Reset. A CRS indication will only be given if the address to be read is vendor ID register. pci_bus_read_dev_vendor_id() knows how to deal with CRS returned 0xFFFF0001 value and will continue polling until a value other than 0xFFFF0001 is returned within a given timeout. Try to discover device presence via CRS first. If device is not found, fall through to old behavior. Signed-off-by: Sinan Kaya --- drivers/pci/pci.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index af0cc34..c853551 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -3821,17 +3821,36 @@ static void pci_flr_wait(struct pci_dev *dev) { int i = 0; u32 id; + bool ret; + + /* + * Don't touch the HW before waiting 100ms. HW has to finish + * within 100ms according to PCI Express Base Specification + * Revision 3.1 Section 6.6.2: Function-Level Reset (FLR). + */ + msleep(100); + + if (pci_bus_read_config_dword(dev->bus, dev->devfn, PCI_VENDOR_ID, + &id)) + return; + + /* See if we can find a device via CRS first. */ + if ((id & 0xffff) == 0x0001) { + ret = pci_bus_wait_crs(dev->bus, dev->devfn, &id, 60000); + if (ret) + return; + } do { msleep(100); pci_read_config_dword(dev, PCI_COMMAND, &id); - } while (i++ < 10 && id == ~0); + } while (i++ < 9 && id == ~0); if (id == ~0) dev_warn(&dev->dev, "Failed to return from FLR\n"); else if (i > 1) dev_info(&dev->dev, "Required additional %dms to return from FLR\n", - (i - 1) * 100); + i * 100); } /**