From patchwork Tue Oct 27 16:47:46 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 7498401 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id E6D5C9F327 for ; Tue, 27 Oct 2015 16:48:37 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id F30DE205F5 for ; Tue, 27 Oct 2015 16:48:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0DCB82042C for ; Tue, 27 Oct 2015 16:48:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754555AbbJ0Qse (ORCPT ); Tue, 27 Oct 2015 12:48:34 -0400 Received: from [193.47.165.129] ([193.47.165.129]:51168 "EHLO mellanox.co.il" rhost-flags-FAIL-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1754539AbbJ0Qse (ORCPT ); Tue, 27 Oct 2015 12:48:34 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from sagig@mellanox.com) with ESMTPS (AES256-SHA encrypted); 27 Oct 2015 18:47:52 +0200 Received: from r-vnc05.mtr.labs.mlnx (r-vnc05.mtr.labs.mlnx [10.208.0.115]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id t9RGlqjW006973; Tue, 27 Oct 2015 18:47:52 +0200 Received: from r-vnc05.mtr.labs.mlnx (localhost [127.0.0.1]) by r-vnc05.mtr.labs.mlnx (8.14.4/8.14.4) with ESMTP id t9RGlqV3001551; Tue, 27 Oct 2015 18:47:52 +0200 Received: (from sagig@localhost) by r-vnc05.mtr.labs.mlnx (8.14.4/8.14.4/Submit) id t9RGlquV001550; Tue, 27 Oct 2015 18:47:52 +0200 From: Sagi Grimberg To: linux-rdma@vger.kernel.org, target-devel@vger.kernel.org Cc: Steve Wise , "Nicholas A. Bellinger" , Or Gerlitz , Doug Ledford Subject: [PATCH v1 1/2] mlx4: Expose correct max_sge_rd limit Date: Tue, 27 Oct 2015 18:47:46 +0200 Message-Id: <1445964467-1497-2-git-send-email-sagig@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1445964467-1497-1-git-send-email-sagig@mellanox.com> References: <1445964467-1497-1-git-send-email-sagig@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP mlx4 devices (ConnectX-2, ConnectX-3) has a limitation where rdma read work queue entries cannot exceed 512 bytes. A rdma_read wqe needs to fit in 512 bytes: - wqe control segment (16 bytes) - rdma segment (12 bytes) - scatter elements (16 bytes each) So max_sge_rd should be: (512 - 16 - 12) / 16 = 30. Signed-off-by: Sagi Grimberg --- drivers/infiniband/hw/mlx4/main.c | 2 +- include/linux/mlx4/device.h | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 3889723..d8453f1 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -499,7 +499,7 @@ static int mlx4_ib_init_device_flags(struct ib_device *ibdev) ibdev->max_qp_wr = dev->dev->caps.max_wqes - MLX4_IB_SQ_MAX_SPARE; ibdev->max_sge = min(dev->dev->caps.max_sq_sg, dev->dev->caps.max_rq_sg); - ibdev->max_sge_rd = ibdev->max_sge; + ibdev->max_sge_rd = MLX4_MAX_SGE_RD; ibdev->max_cq = dev->dev->quotas.cq; ibdev->max_cqe = dev->dev->caps.max_cqes; ibdev->max_mr = dev->dev->quotas.mpt; diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index baad4cb..90c12f0 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -425,6 +425,17 @@ enum { }; enum { + /* + * Max wqe size for rdma read is 512 bytes, so this + * limits our max_sge_rd as the wqe needs to fit: + * - ctrl segment (16 bytes) + * - rdma segment (12 bytes) + * - scatter elements (16 bytes each) + */ + MLX4_MAX_SGE_RD = (512 - 16 - 12) / 16 +}; + +enum { MLX4_DEV_PMC_SUBTYPE_GUID_INFO = 0x14, MLX4_DEV_PMC_SUBTYPE_PORT_INFO = 0x15, MLX4_DEV_PMC_SUBTYPE_PKEY_TABLE = 0x16,