From patchwork Tue Mar 7 15:04:48 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Rados=C5=82aw_Biernacki?= X-Patchwork-Id: 9609077 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 408C6602B4 for ; Tue, 7 Mar 2017 15:06:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D45228514 for ; Tue, 7 Mar 2017 15:06:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 202C22851D; Tue, 7 Mar 2017 15:06:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id EC2F128514 for ; Tue, 7 Mar 2017 15:06:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=76S74F5yU3zi+j5wRH7i5VX4vu12HwKPGdiyH/Ch8UQ=; b=Z85 eGc/DvpBbMG8dEyGS8ILkjXTVKgjvRRQV3Vf+znsbP4RvKAAbx9tJhnOAdwKT4EA+7lXj7xSWz3/i dE9Vx3Hj0uDQY/Y2wuTGsG4WGRyrAtqpBUT9HoXgKHjOWB/VGGjYmGLRALwXdGt6bAQd7AHDdsEyX 6G98K7m1iLLhjuhK/FyDB3u53k3TIQHlpSPPW/c7BvSEV+ReNA5xx47RtujWIp6KD8o/OVpoAzkaJ qZIoIJj/OxBMGGXuXo0piFmKSqX7Vt4fJ1oiXtgunTg/S4m9Zfjxtl2zUhqT7sZAJKg8YSpioDvqM 0cLDd7jGn1VNqfczmi42Y46tFMJ/YHw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1clGh0-00019w-Hq; Tue, 07 Mar 2017 15:06:26 +0000 Received: from mail-it0-x22b.google.com ([2607:f8b0:4001:c0b::22b]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1clGgZ-0000xa-Od for linux-arm-kernel@lists.infradead.org; Tue, 07 Mar 2017 15:06:02 +0000 Received: by mail-it0-x22b.google.com with SMTP id w124so7561937itb.0 for ; Tue, 07 Mar 2017 07:05:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id; bh=kJTZdy3xQN5dnZo1dhDS9tk4sMWA2jnwAUXy2r1q52w=; b=Dc3F+F5zZgtb1+wgFbwtgusZ9M9e2vaKKzM7HU0ACXIYKW7s76YpGBAhghcH+mViCL Dq9w33lxU8ftQreFsjT9eqAW+Q0d3L41A+ttoyHV2HbpQgHMCYZ+LFP7TJFa3Srv4PEJ AkoR1j9YqVgo69SqjQtQ3NCPwPgFaLwEO59aGqkGr55umFXqueow2pnftcCsOfEnhFAP aKWzsjVOU+sjAS+FsEt/OJMBgiN/vxz9Xxt17cXQ/jy+k+pbp7ebYRkDKmcx5sBRcRN0 Gg9QxKIoMNG5Lw4z/JcbnrbAhi+MA2ls0BRLa1XG4tgdHJJYDVg19H1g+5g+4OYDljhc 2pWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=kJTZdy3xQN5dnZo1dhDS9tk4sMWA2jnwAUXy2r1q52w=; b=SlF7vkg1InPsIeB8X4lp43RGgczpTrpddScuO6MC+p0Dpoe6U+Mgnjn+dsgMQItkRA rPN6j8MQyunYQiMblgrPzaWkhceDnlvlTZLujryXT7YPuhD1WjHief//sEfvSzq9cYdJ XP7OyLO201QjLo9PUlixGTxprCQTg0gHBLGFsX+csh9lPW8vDUvMtB8+1hdNMmXoMvL4 ajkQbceW7WFQotWO66Ml9ROQpQceqvykkyl3PM6FA36p14TZDo4HmsYC7FTYnYLevOqi 12gjyQx9FpliBZbOnpU4J1NSJ0/GzjZyv7JhtcXlhTk641jlTpv/yMtCrQRiIRlnO+bk /Zkg== X-Gm-Message-State: AMke39mOhhlghEy/6/uszat+B2XKYDG1lhryt8ZcZinvW95fVaFP8489czMB6UPCHHdxOQ== X-Received: by 10.36.199.66 with SMTP id t63mr1241031itg.10.1488899138154; Tue, 07 Mar 2017 07:05:38 -0800 (PST) Received: from rad-H81M-S1.semihalf.local (31-172-191-173.noc.fibertech.net.pl. [31.172.191.173]) by smtp.gmail.com with ESMTPSA id h11sm224838ioa.43.2017.03.07.07.05.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 07 Mar 2017 07:05:36 -0800 (PST) From: Radoslaw Biernacki To: Bjorn Helgaas , Sunil.Goutham@cavium.com, Robert Richter , David Daney , Robert Richter , David Miller Subject: [PATCH] PCI: Add pci reset quirk for Cavium VNIC Date: Tue, 7 Mar 2017 16:04:48 +0100 Message-Id: <1488899088-10194-1-git-send-email-rad@semihalf.com> X-Mailer: git-send-email 1.9.1 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170307_070600_008753_E32CFB9C X-CRM114-Status: GOOD ( 13.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Linu.Cherian@cavium.com, Radoslaw Biernacki , Maciej Czekaj , Tomasz Nowicki , Radoslaw Biernacki , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-pci@vger.kernel.org, Marcin Wojtas , linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP From: Radoslaw Biernacki PCI reset quirk is needed for Cavium Function NIC since it does not handle a function level reset. This cause problems when VNIC is used from userspace by vfio. If application (or VM) does not stop the VNIC queues, HW may cause overwrite of memory locations when next application that use it will establish new SMMU mappings. More likely HW it will cause SMMU exception, when network packet will came before new SMMU mappings will be made. Signed-off-by: Radoslaw Biernacki Reviewed-by: Sunil Goutham --- drivers/pci/quirks.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index f754453..f7cdbe5 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3783,10 +3783,98 @@ static int reset_chelsio_generic_dev(struct pci_dev *dev, int probe) return 0; } +#define CAVIUM_VNIC_QSET_RQ_0_7_CFG (0x010600) +#define CAVIUM_VNIC_QSET_CQ_0_7_CFG (0x010400) +#define CAVIUM_VNIC_QSET_CQ_0_7_CFG2 (0x010408) +#define CAVIUM_VNIC_QSET_SQ_0_7_CFG (0x010800) +#define CAVIUM_VNIC_QSET_SQ_0_7_STATUS (0x010840) +#define CAVIUM_VNIC_QSET_RBDR_0_1_CFG (0x010C00) +#define CAVIUM_VNIC_QSET_RBDR_0_1_STATUS0 (0x010C40) + +#define CAVIUM_VNIC_Q_SHIFT (18) +#define CAVIUM_VNIC_CQ_RESET (1ULL << 41) +#define CAVIUM_VNIC_SQ_RESET (1ULL << 17) +#define CAVIUM_VNIC_RBDR_RESET (1ULL << 43) +#define CAVIUM_VNIC_RBDR_FIFO_STATE_SHIFT (62) + +/* Poll a register for a specific value */ +static int cavium_vnic_poll(struct pci_dev *pdev, + void __iomem *addr, size_t bit_pos, + size_t bits, u64 val) +{ + u64 bit_mask; + u64 reg_val; + size_t timeout = 10; + + bit_mask = (1ULL << bits) - 1; + bit_mask = (bit_mask << bit_pos); + + while (timeout) { + reg_val = readq(addr); + if (((reg_val & bit_mask) >> bit_pos) == val) + return 0; + usleep_range(1000, 2000); + timeout--; + } + dev_err(&pdev->dev, "Poll on addr %p failed\n", addr); + return -1; +} + +static int cavium_vnic_reset(struct pci_dev *pdev, int probe) +{ + size_t qidx; + void __iomem *bar_base; + void __iomem *qset_base; + + bar_base = pci_iomap(pdev, 0, 0); + if (!bar_base) + return -ENOMEM; + + /* For each of 8 RQ/CQ/SQ (queues) in VF */ + for (qidx = 0; qidx < 8; qidx++) { + /* Disable receive queue */ + qset_base = bar_base + (qidx << CAVIUM_VNIC_Q_SHIFT); + writeq(0, qset_base + CAVIUM_VNIC_QSET_RQ_0_7_CFG); + + /* Disable timer threshold (doesn't get reset upon CQ reset */ + writeq(0, qset_base + CAVIUM_VNIC_QSET_CQ_0_7_CFG2); + /* Disable completion queue */ + writeq(0, qset_base + CAVIUM_VNIC_QSET_CQ_0_7_CFG); + /* Reset completion queue */ + writeq(CAVIUM_VNIC_CQ_RESET, + qset_base + CAVIUM_VNIC_QSET_CQ_0_7_CFG); + + /* Disable send queue */ + writeq(0, qset_base + CAVIUM_VNIC_QSET_SQ_0_7_CFG); + /* Reset send queue */ + writeq(CAVIUM_VNIC_SQ_RESET, + qset_base + CAVIUM_VNIC_QSET_SQ_0_7_CFG); + } + + /* Reset and disable both RBDR's */ + for (qidx = 0; qidx < 2; qidx++) { + qset_base = bar_base + + (qidx << CAVIUM_VNIC_Q_SHIFT); + writeq(CAVIUM_VNIC_RBDR_RESET, + qset_base + CAVIUM_VNIC_QSET_RBDR_0_1_CFG); + writeq(0, qset_base + CAVIUM_VNIC_QSET_RBDR_0_1_CFG); + if (cavium_vnic_poll(pdev, qset_base + + CAVIUM_VNIC_QSET_RBDR_0_1_STATUS0, + CAVIUM_VNIC_RBDR_FIFO_STATE_SHIFT, + 2, 0x00)) + dev_err(&pdev->dev, "Timeout on RBDR reset sequence"); + } + + pci_iounmap(pdev, bar_base); + return 0; +} + #define PCI_DEVICE_ID_INTEL_82599_SFP_VF 0x10ed #define PCI_DEVICE_ID_INTEL_IVB_M_VGA 0x0156 #define PCI_DEVICE_ID_INTEL_IVB_M2_VGA 0x0166 +#define PCI_DEVICE_ID_CAVIUM_NIC_VF 0xA034 + static const struct pci_dev_reset_methods pci_dev_reset_methods[] = { { PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_82599_SFP_VF, reset_intel_82599_sfp_virtfn }, @@ -3796,6 +3884,8 @@ static int reset_chelsio_generic_dev(struct pci_dev *dev, int probe) reset_ivb_igd }, { PCI_VENDOR_ID_CHELSIO, PCI_ANY_ID, reset_chelsio_generic_dev }, + { PCI_VENDOR_ID_CAVIUM, PCI_DEVICE_ID_CAVIUM_NIC_VF, + cavium_vnic_reset }, { 0 } };