From patchwork Sun Nov 19 19:21:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 10065761 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C1811602B8 for ; Sun, 19 Nov 2017 19:22:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B091728F12 for ; Sun, 19 Nov 2017 19:22:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A58C02906E; Sun, 19 Nov 2017 19:22:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EBB0728F12 for ; Sun, 19 Nov 2017 19:22:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751043AbdKSTWA (ORCPT ); Sun, 19 Nov 2017 14:22:00 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:54786 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750954AbdKSTWA (ORCPT ); Sun, 19 Nov 2017 14:22:00 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with ESMTPS (AES256-SHA encrypted); 19 Nov 2017 21:21:54 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id vAJJLsLW018108; Sun, 19 Nov 2017 21:21:54 +0200 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id vAJJLs9d006190; Sun, 19 Nov 2017 21:21:54 +0200 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id vAJJLsmg006189; Sun, 19 Nov 2017 21:21:54 +0200 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: yishaih@mellanox.com, noaos@mellanox.com, majd@mellanox.com Subject: [PATCH rdma-core 1/2] mlx5: Report Multi-Packet RQ capabilities through mlx5 direct verbs Date: Sun, 19 Nov 2017 21:21:07 +0200 Message-Id: <1511119268-5934-2-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1511119268-5934-1-git-send-email-yishaih@mellanox.com> References: <1511119268-5934-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Noa Osherovich A Multi-Packet RQ is a receive queue where multiple packets are written to the same WQE. Each message starts in the beginning of a stride. The total size of the scatter elements of each WQE is determined upon RQ creation and all the posted WQEs should meet the determined size. A Multi-Packet RQ reduces the number of needed post-recv operations thus increasing performance. It reduces memory footprint by allowing each packet to consume a different number of strides instead of the whole WR. Signed-off-by: Noa Osherovich Reviewed-by: Yishai Hadas --- providers/mlx5/man/mlx5dv_query_device.3 | 14 +++++++++++++- providers/mlx5/mlx5-abi.h | 6 ++++++ providers/mlx5/mlx5.c | 5 +++++ providers/mlx5/mlx5.h | 1 + providers/mlx5/mlx5dv.h | 12 +++++++++++- providers/mlx5/verbs.c | 1 + 6 files changed, 37 insertions(+), 2 deletions(-) diff --git a/providers/mlx5/man/mlx5dv_query_device.3 b/providers/mlx5/man/mlx5dv_query_device.3 index eca6fc1..f7cfdc0 100644 --- a/providers/mlx5/man/mlx5dv_query_device.3 +++ b/providers/mlx5/man/mlx5dv_query_device.3 @@ -29,6 +29,17 @@ uint32_t supported_qpts; }; .PP .nf +struct mlx5dv_striding_rq_caps { +.in +8 +uint32_t min_single_stride_log_num_of_bytes; /* min log size of each stride */ +uint32_t max_single_stride_log_num_of_bytes; /* max log size of each stride */ +uint32_t min_single_wqe_log_num_of_strides; /* min log number of strides per WQE */ +uint32_t max_single_wqe_log_num_of_strides; /* max log number of strides per WQE */ +uint32_t supported_qpts; +.in -8 +}; +.PP +.nf struct mlx5dv_context { .in +8 uint8_t version; @@ -59,7 +70,8 @@ enum mlx5dv_context_comp_mask { .in +8 MLX5DV_CONTEXT_MASK_CQE_COMPRESION = 1 << 0, MLX5DV_CONTEXT_MASK_SWP = 1 << 1, -MLX5DV_CONTEXT_MASK_RESERVED = 1 << 2, +MLX5DV_CONTEXT_MASK_STRIDING_RQ = 1 << 2, +MLX5DV_CONTEXT_MASK_RESERVED = 1 << 3, .in -8 }; diff --git a/providers/mlx5/mlx5-abi.h b/providers/mlx5/mlx5-abi.h index e1f5114..b569bd4 100644 --- a/providers/mlx5/mlx5-abi.h +++ b/providers/mlx5/mlx5-abi.h @@ -288,6 +288,11 @@ enum mlx5_query_dev_resp_flags { MLX5_QUERY_DEV_RESP_FLAGS_CQE_128B_PAD = 1 << 1, }; +struct mlx5_striding_rq_caps { + struct mlx5dv_striding_rq_caps caps; + __u32 reserved; +}; + struct mlx5_query_device_ex_resp { struct ibv_query_device_resp_ex ibv_resp; __u32 comp_mask; @@ -299,6 +304,7 @@ struct mlx5_query_device_ex_resp { __u32 support_multi_pkt_send_wqe; __u32 flags; /* Use enum mlx5_query_dev_resp_flags */ struct mlx5dv_sw_parsing_caps sw_parsing_caps; + struct mlx5_striding_rq_caps striding_rq_caps; }; #endif /* MLX5_ABI_H */ diff --git a/providers/mlx5/mlx5.c b/providers/mlx5/mlx5.c index 70afbd4..36b47d7 100644 --- a/providers/mlx5/mlx5.c +++ b/providers/mlx5/mlx5.c @@ -641,6 +641,11 @@ int mlx5dv_query_device(struct ibv_context *ctx_in, comp_mask_out |= MLX5DV_CONTEXT_MASK_SWP; } + if (attrs_out->comp_mask & MLX5DV_CONTEXT_MASK_STRIDING_RQ) { + attrs_out->striding_rq_caps = mctx->striding_rq_caps; + comp_mask_out |= MLX5DV_CONTEXT_MASK_STRIDING_RQ; + } + attrs_out->comp_mask = comp_mask_out; return 0; diff --git a/providers/mlx5/mlx5.h b/providers/mlx5/mlx5.h index b4782dd..7c85ab6 100644 --- a/providers/mlx5/mlx5.h +++ b/providers/mlx5/mlx5.h @@ -292,6 +292,7 @@ struct mlx5_context { struct mlx5dv_cqe_comp_caps cqe_comp_caps; struct mlx5dv_ctx_allocators extern_alloc; struct mlx5dv_sw_parsing_caps sw_parsing_caps; + struct mlx5dv_striding_rq_caps striding_rq_caps; }; struct mlx5_bitmap { diff --git a/providers/mlx5/mlx5dv.h b/providers/mlx5/mlx5dv.h index 935ff54..3566bcb 100644 --- a/providers/mlx5/mlx5dv.h +++ b/providers/mlx5/mlx5dv.h @@ -60,7 +60,8 @@ enum { enum mlx5dv_context_comp_mask { MLX5DV_CONTEXT_MASK_CQE_COMPRESION = 1 << 0, MLX5DV_CONTEXT_MASK_SWP = 1 << 1, - MLX5DV_CONTEXT_MASK_RESERVED = 1 << 2, + MLX5DV_CONTEXT_MASK_STRIDING_RQ = 1 << 2, + MLX5DV_CONTEXT_MASK_RESERVED = 1 << 3, }; struct mlx5dv_cqe_comp_caps { @@ -73,6 +74,14 @@ struct mlx5dv_sw_parsing_caps { uint32_t supported_qpts; }; +struct mlx5dv_striding_rq_caps { + uint32_t min_single_stride_log_num_of_bytes; + uint32_t max_single_stride_log_num_of_bytes; + uint32_t min_single_wqe_log_num_of_strides; + uint32_t max_single_wqe_log_num_of_strides; + uint32_t supported_qpts; +}; + /* * Direct verbs device-specific attributes */ @@ -82,6 +91,7 @@ struct mlx5dv_context { uint64_t comp_mask; struct mlx5dv_cqe_comp_caps cqe_comp_caps; struct mlx5dv_sw_parsing_caps sw_parsing_caps; + struct mlx5dv_striding_rq_caps striding_rq_caps; }; enum mlx5dv_context_flags { diff --git a/providers/mlx5/verbs.c b/providers/mlx5/verbs.c index 19fc947..478d9a1 100644 --- a/providers/mlx5/verbs.c +++ b/providers/mlx5/verbs.c @@ -2156,6 +2156,7 @@ int mlx5_query_device_ex(struct ibv_context *context, mctx->cqe_comp_caps = resp.cqe_comp_caps; mctx->sw_parsing_caps = resp.sw_parsing_caps; + mctx->striding_rq_caps = resp.striding_rq_caps.caps; if (resp.flags & MLX5_QUERY_DEV_RESP_FLAGS_CQE_128B_COMP) mctx->vendor_cap_flags |= MLX5_VENDOR_CAP_FLAGS_CQE_128B_COMP;