From patchwork Thu Mar 31 16:03:25 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 8714461 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 52F6A9F30C for ; Thu, 31 Mar 2016 16:03:39 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id BB326202A1 for ; Thu, 31 Mar 2016 16:03:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 25FE5201EF for ; Thu, 31 Mar 2016 16:03:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751813AbcCaQDa (ORCPT ); Thu, 31 Mar 2016 12:03:30 -0400 Received: from mail-wm0-f66.google.com ([74.125.82.66]:33200 "EHLO mail-wm0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751359AbcCaQD3 (ORCPT ); Thu, 31 Mar 2016 12:03:29 -0400 Received: by mail-wm0-f66.google.com with SMTP id i204so23156730wmd.0 for ; Thu, 31 Mar 2016 09:03:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id; bh=eXbc4zWz2cHcl8yHVhO28mS4jatRx5pl+zGSXuFka+Q=; b=cUoFSk/dA02xe/cnQ+9vsvWMk+oHL0505URNNTN3GaG1L0ejHZsM6pOQmHK8QAU9o4 5Fz8XOCTFx78z+hYim8TBe8eHBJkfU4ogAuwXyBxaqeJct0++bmaRE3Jeyx1jC+mtn9k Tok8fgwmtOOxVRE/6W4oQdWrMW6aMygnmmScPcLEXH6+NFJ5XodjcHwLYWxJTlTjnArP vNwkZrL++mX82O/NurZSw42N7x8qe0ii8yBaMhgL11reSJ4LX9bpOD83FQXKbMjI7/jh 1ENRQ/V5450GYytZQOWDr7rWC3ZxtHZN5nWz2chmI3/7hMZ3kzuu7Vp3nAbStlcSwB3J cpqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id; bh=eXbc4zWz2cHcl8yHVhO28mS4jatRx5pl+zGSXuFka+Q=; b=S19srssFJl6AK+J2R+gyBdtnSqrfBEA4EgwNgntSPkJQ+i6BT7xoqr5xcNgsEmkgfs r2fT9nGvqk6u4TKjOgduRwrv+kcp930mZlzHL7AoAQDumcCt4B3WqFBfpa4hRYgfqtIJ xksx1MqN8nmebzO9gyAKA7SCfnnF1EUwLwbhokSVPOIpPZaRFnryjEfuvWS3gOrebcGI TUBt3Em9/JCEQtBaCggs/AHKqi2lAmeGt5MlG8++utTNtykXpBhtlnzAw4uNswhddy3h cbcQnICxOT8bqjwDJg9oOqO6w42OSGVeVkJM4nmreJOyLCG4xA9te/539w/VuV9yUeCz nk7w== X-Gm-Message-State: AD7BkJKxbu/KwfhVMvqRTv86tpwEAsh5hzeHtaAdyBZ8kzWpTZXUbZt7u6UYKwgEqZOsCQ== X-Received: by 10.194.77.193 with SMTP id u1mr4059058wjw.73.1459440207995; Thu, 31 Mar 2016 09:03:27 -0700 (PDT) Received: from sagi-Latitude-E7470 (bzq-82-81-101-184.red.bezeqint.net. [82.81.101.184]) by smtp.gmail.com with ESMTPSA id 188sm14761416wml.19.2016.03.31.09.03.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 31 Mar 2016 09:03:27 -0700 (PDT) From: Sagi Grimberg X-Google-Original-From: Sagi Grimberg Received: by sagi-Latitude-E7470 (Postfix, from userid 1000) id E4985A62307; Thu, 31 Mar 2016 19:03:25 +0300 (IDT) To: linux-rdma@vger.kernel.org Cc: Matan Barak , Leon Romanovsky Subject: [PATCH] IB/mlx5: Expose correct max_sge_rd limit Date: Thu, 31 Mar 2016 19:03:25 +0300 Message-Id: <1459440205-25208-1-git-send-email-sagig@grimberg.me> X-Mailer: git-send-email 1.9.1 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-7.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP mlx5 devices (Connect-IB, ConnectX-4, ConnectX-4-LX) has a limitation where rdma read work queue entries cannot exceed 512 bytes. A rdma_read wqe needs to fit in 512 bytes: - wqe control segment (16 bytes) - rdma segment (16 bytes) - scatter elements (16 bytes each) So max_sge_rd should be: (512 - 16 - 16) / 16 = 30. Reported-by: Christoph Hellwig Tested-by: Christoph Hellwig Signed-off-by: Sagi Grimberg Signed-off-by: Leon Romanovsky --- drivers/infiniband/hw/mlx5/main.c | 2 +- include/linux/mlx5/device.h | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index 03c418ccbc98..ed9cefa1f6f1 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -517,7 +517,7 @@ static int mlx5_ib_query_device(struct ib_device *ibdev, sizeof(struct mlx5_wqe_ctrl_seg)) / sizeof(struct mlx5_wqe_data_seg); props->max_sge = min(max_rq_sg, max_sq_sg); - props->max_sge_rd = props->max_sge; + props->max_sge_rd = MLX5_MAX_SGE_RD; props->max_cq = 1 << MLX5_CAP_GEN(mdev, log_max_cq); props->max_cqe = (1 << MLX5_CAP_GEN(mdev, log_max_cq_sz)) - 1; props->max_mr = 1 << MLX5_CAP_GEN(mdev, log_max_mkey); diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h index 987764afa65c..f8b83792939b 100644 --- a/include/linux/mlx5/device.h +++ b/include/linux/mlx5/device.h @@ -363,6 +363,17 @@ enum { MLX5_CAP_OFF_CMDIF_CSUM = 46, }; +enum { + /* + * Max wqe size for rdma read is 512 bytes, so this + * limits our max_sge_rd as the wqe needs to fit: + * - ctrl segment (16 bytes) + * - rdma segment (16 bytes) + * - scatter elements (16 bytes each) + */ + MLX5_MAX_SGE_RD = (512 - 16 - 16) / 16 +}; + struct mlx5_inbox_hdr { __be16 opcode; u8 rsvd[4];