From patchwork Wed Aug 9 00:57:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sinan Kaya X-Patchwork-Id: 9889361 X-Patchwork-Delegate: bhelgaas@google.com Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B969C601EB for ; Wed, 9 Aug 2017 00:57:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAB98289B5 for ; Wed, 9 Aug 2017 00:57:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9F31C289C0; Wed, 9 Aug 2017 00:57:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7580A289BC for ; Wed, 9 Aug 2017 00:57:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752439AbdHIA5d (ORCPT ); Tue, 8 Aug 2017 20:57:33 -0400 Received: from smtp.codeaurora.org ([198.145.29.96]:51068 "EHLO smtp.codeaurora.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752319AbdHIA5c (ORCPT ); Tue, 8 Aug 2017 20:57:32 -0400 Received: by smtp.codeaurora.org (Postfix, from userid 1000) id D5FF960398; Wed, 9 Aug 2017 00:57:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1502240251; bh=ubhUWLjJZ+eYB02aTL2/riAzMK/vNDPmvUgvsAz+xwI=; h=From:To:Cc:Subject:Date:From; b=d32mqrPtD4MPIuJFBcF38ORqxqMdFiCaT3p5NWn4MQy6JxrCtpHhezSlS1KwXz0iC V/1IjcE96kj1V3ot48hSYNmtOY6EJh0wXxdadDrZMHmDR01ivL4i5uRgeGoqk2mXOX iXPfNCQGKo6pBaoCoEKs8mIlmPfO7oTmTZBQcnes= Received: from drakthul.qualcomm.com (global_nat1_iad_fw.qualcomm.com [129.46.232.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: okaya@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id 5544B6014C; Wed, 9 Aug 2017 00:57:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1502240251; bh=ubhUWLjJZ+eYB02aTL2/riAzMK/vNDPmvUgvsAz+xwI=; h=From:To:Cc:Subject:Date:From; b=d32mqrPtD4MPIuJFBcF38ORqxqMdFiCaT3p5NWn4MQy6JxrCtpHhezSlS1KwXz0iC V/1IjcE96kj1V3ot48hSYNmtOY6EJh0wXxdadDrZMHmDR01ivL4i5uRgeGoqk2mXOX iXPfNCQGKo6pBaoCoEKs8mIlmPfO7oTmTZBQcnes= DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 5544B6014C Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=okaya@codeaurora.org From: Sinan Kaya To: linux-pci@vger.kernel.org, timur@codeaurora.org, alex.williamson@redhat.com Cc: linux-arm-msm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sinan Kaya , Bjorn Helgaas , linux-kernel@vger.kernel.org Subject: [PATCH V9 1/2] PCI: handle CRS returned by device after FLR Date: Tue, 8 Aug 2017 20:57:24 -0400 Message-Id: <1502240245-11714-1-git-send-email-okaya@codeaurora.org> X-Mailer: git-send-email 1.9.1 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Sporadic reset issues have been observed with Intel 750 NVMe drive by writing to the reset file in sysfs in a loop. The sequence of events observed is as follows: - perform a Function Level Reset (FLR) - sleep up to 1000ms total - read ~0 from PCI_COMMAND - warn that the device didn't return from FLR - touch the device before it's ready An endpoint is allowed to issue Configuration Request Retry Status (CRS) following a FLR request to indicate that it is not ready to accept new requests. CRS is defined in PCIe r3.1, sec 2.3.1. Request Handling Rules and CRS usage in FLR context is mentioned in PCIe r3.1a, sec 6.6.2. Function-Level Reset. A CRS indication will only be given if the address to be read is vendor ID register. pci_bus_read_dev_vendor_id() knows how to deal with CRS returned 0xFFFF0001 value and will continue polling until a value other than 0xFFFF0001 is returned within a given timeout. Try to discover device presence via CRS first. If device is not found, fall through to old behavior. Signed-off-by: Sinan Kaya --- drivers/pci/pci.c | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index af0cc34..4366299 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -3821,17 +3821,39 @@ static void pci_flr_wait(struct pci_dev *dev) { int i = 0; u32 id; + bool ret; + + /* + * Don't touch the HW before waiting 100ms. HW has to finish + * within 100ms according to PCI Express Base Specification + * Revision 3.1 Section 6.6.2: Function-Level Reset (FLR). + */ + msleep(100); + + /* + * See if we can find a device via CRS first. Physical functions + * return from here if found, Virtual functions fall through as + * they return ~0 on vendor id read once CRS is completed. + */ + ret = pci_bus_read_dev_vendor_id(dev->bus, dev->devfn, &id, + 60000); + if (ret) + return; + + pci_read_config_dword(dev, PCI_COMMAND, &id); + if (id != ~0) + return; do { msleep(100); pci_read_config_dword(dev, PCI_COMMAND, &id); - } while (i++ < 10 && id == ~0); + } while (i++ < 9 && id == ~0); if (id == ~0) dev_warn(&dev->dev, "Failed to return from FLR\n"); else if (i > 1) dev_info(&dev->dev, "Required additional %dms to return from FLR\n", - (i - 1) * 100); + i * 100); } /**