From patchwork Mon Jan 9 09:38:19 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shamir Rabinovitch X-Patchwork-Id: 9504257 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4FC6B60757 for ; Mon, 9 Jan 2017 09:38:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2BAF5281D2 for ; Mon, 9 Jan 2017 09:38:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1E04D28413; Mon, 9 Jan 2017 09:38:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 41B5C281D2 for ; Mon, 9 Jan 2017 09:38:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934008AbdAIJii (ORCPT ); Mon, 9 Jan 2017 04:38:38 -0500 Received: from userp1040.oracle.com ([156.151.31.81]:51549 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754306AbdAIJih (ORCPT ); Mon, 9 Jan 2017 04:38:37 -0500 Received: from userv0021.oracle.com (userv0021.oracle.com [156.151.31.71]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v099cWfS005784 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 9 Jan 2017 09:38:33 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0021.oracle.com (8.14.4/8.14.4) with ESMTP id v099cWPR023157 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 9 Jan 2017 09:38:32 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id v099cVFm023758; Mon, 9 Jan 2017 09:38:32 GMT Received: from shamir-net-srv.us.oracle.com (/10.211.3.142) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 09 Jan 2017 01:38:31 -0800 From: Shamir Rabinovitch To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, shamir.rabinovitch@oracle.com Subject: [PATCH v2] net/mlx4_core: VF probe fail when HW support 64-bit coherent DMA Date: Mon, 9 Jan 2017 04:38:19 -0500 Message-Id: <1483954699-17826-1-git-send-email-shamir.rabinovitch@oracle.com> X-Mailer: git-send-email 1.7.1 X-Source-IP: userv0021.oracle.com [156.151.31.71] Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If IOMMU support 64-bit coherent DMA mlx4_core driver will try to use it and VF probe will fail due to firmware error. Force all mlx4_core VFs coherent DMA to 32-bit only! Kernel log when issue occur: [1383654.766249] mlx4_core 0006:01:00.1: Sending reset [1383654.775971] mlx4_core 0006:01:00.0: Received reset from slave:1 [1383654.788087] mlx4_core 0006:01:00.1: Sending vhcr0 [1383664.318338] mlx4_core 0006:01:00.0: command 0x2e failed: fw status = 0x1 [1383664.318342] mlx4_core 0006:01:00.0: mlx4_master_process_vhcr: Failed reading vhcr ret: 0xfffffffb [1383664.318345] mlx4_core 0006:01:00.0: Failed processing vhcr for slave:1, resetting slave [1383664.318352] mlx4_core 0006:01:00.0: Turn on internal error to force reset, slave=1, cmd=0x5 [1383664.318415] mlx4_core 0006:01:00.0: slave:1 is out of sync, cmd=0x5, last command=0x0, reset is needed [1383664.318418] mlx4_core 0006:01:00.0: Turn on internal error to force reset, slave=1, cmd=0x5 [1383664.318501] mlx4_core 0006:01:00.0: slave:1 is out of sync, cmd=0x5, last command=0x0, reset is needed [1383664.318504] mlx4_core 0006:01:00.0: Turn on internal error to force reset, slave=1, cmd=0x5 [1383664.318513] mlx4_core 0006:01:00.1: HCA minimum page size:1 [1383664.318515] mlx4_core 0006:01:00.1: UAR size:4096 != kernel PAGE_SIZE of 8192 [1383664.318517] mlx4_core 0006:01:00.1: Failed to obtain slave caps Signed-off-by: Shamir Rabinovitch --- Changelog: v1 -> v2: Review comments from Christoph Hellwig . Verified and only VF require 32-bit coherent DMA. PF can still use 64-bit coherent DMA. --- --- drivers/net/ethernet/mellanox/mlx4/main.c | 20 +++++++++++++++++--- 1 files changed, 17 insertions(+), 3 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index bffa6f3..131cbc9 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -3719,9 +3719,23 @@ static int __mlx4_init_one(struct pci_dev *pdev, int pci_dev_data, goto err_release_regions; } } - err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64)); - if (err) { - dev_warn(&pdev->dev, "Warning: couldn't set 64-bit consistent PCI DMA mask\n"); + if (!(pci_dev_data & MLX4_PCI_DEV_IS_VF)) { + err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64)); + if (err) { + dev_warn(&pdev->dev, + "Warning: couldn't set 64-bit consistent PCI DMA mask\n"); + err = pci_set_consistent_dma_mask(pdev, + DMA_BIT_MASK(32)); + if (err) { + dev_err(&pdev->dev, + "Can't set consistent PCI DMA mask, aborting\n"); + goto err_release_regions; + } + } + } else { + /* CX3 firmware 2.11.1280 does not support 64-bit coherent + * DMA for VFs. + */ err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32)); if (err) { dev_err(&pdev->dev, "Can't set consistent PCI DMA mask, aborting\n");