From patchwork Sun Dec 20 11:23:10 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 7891781 Return-Path: X-Original-To: patchwork-linux-rdma@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 9EF5C9F350 for ; Sun, 20 Dec 2015 11:24:14 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id A7BFF2047D for ; Sun, 20 Dec 2015 11:24:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A6AFF2047B for ; Sun, 20 Dec 2015 11:24:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932563AbbLTLYL (ORCPT ); Sun, 20 Dec 2015 06:24:11 -0500 Received: from mail-wm0-f54.google.com ([74.125.82.54]:37832 "EHLO mail-wm0-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932506AbbLTLYK (ORCPT ); Sun, 20 Dec 2015 06:24:10 -0500 Received: by mail-wm0-f54.google.com with SMTP id p187so37083196wmp.0 for ; Sun, 20 Dec 2015 03:24:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leon-nu.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=dqXxbGijMhTv0PNmqlCv6OL98HNcl52ZsnnQh3p2LCU=; b=QYUNz61+d/VAR91debCwO3cMM8MPoQCT3orPBSMQ3++6Bhh0XOJR5Cc2poxPE92bUM oDUPqlTnvQSiuMXIQosiiSnDVw4Be0dW0TN707//qWO5NLVRKyDhCe36VLZ4hEFHVEjV 1WFJPxtWmlmXaAszjpT31fArFe3+OvwGJoKWkEVL9B1BCDco6gCyYk5eGimt6kRinf63 TRP7CDpqzWK24umuWyfrv3oawLAPrCHzaEdZ+yMJtySlvGeBYWEg2FkoI0KKzlDbdbP/ 6enoQletb+IKAS37a/4mYa/Da+533w+8HAuml2QDfJUaSSyd2KrVSqSX1lXVU0pgBqRE wcUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=dqXxbGijMhTv0PNmqlCv6OL98HNcl52ZsnnQh3p2LCU=; b=EWoSd9Y/syr7ULVPLclT5e+wIHLR1kanpeWG8L9VfBfGOQVODllFZr2G7zCrOe9cTN x0lbXOZ+m6jlseeOdjQ1qGmx6Ekd/mN9L4czMvYRiPpI8FvHDgZhnw1ntLRgHeqAKlke l1G+KmoN6o3XqDaTB3QxZ7i5db1OoYh68B8O7E5Euq8eBnB1841JiWbjY66diJEf/Rrm 2g79/aQCClQaDXSy/QOd3i4JKUBLe6hrl7EkT+Lw3X5L2oRYkhUJJD8i2e8bcm6jmhJM n3ECuiIYoNM28H3++ZET6BV62eY/hq7BfzmUHL9josWgsSK5XBHd73UxArSilYuxfvfa hmiw== X-Gm-Message-State: ALoCoQm0PbkH64bJfzwd4+kzLdLLlQVxalTRm5Xkrgl7YIFf10dxZtpB4a9KYGOB6cFC43ZrRmhppY+yrg2urEJCCy8UVYX7eg== X-Received: by 10.28.139.206 with SMTP id n197mr13251394wmd.20.1450610649496; Sun, 20 Dec 2015 03:24:09 -0800 (PST) Received: from localhost ([213.57.247.249]) by smtp.gmail.com with ESMTPSA id m190sm15011194wma.5.2015.12.20.03.24.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 20 Dec 2015 03:24:08 -0800 (PST) From: Leon Romanovsky To: eli@mellanox.com Cc: linux-rdma@vger.kernel.org, Leon Romanovsky Subject: [PATCH libmlx5 2/2] libmlx5: Add CQ ignore overrun creation flag Date: Sun, 20 Dec 2015 13:23:10 +0200 Message-Id: <1450610590-17361-3-git-send-email-leon@leon.nu> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1450610590-17361-1-git-send-email-leon@leon.nu> References: <1450610590-17361-1-git-send-email-leon@leon.nu> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Leon Romanovsky In order to use CQ for cross-channel operations, the user needs to configure CQ to ignore overrun creation flag. This patch adds support of it to libmlx5 library. Signed-off-by: Leon Romanovsky --- src/mlx5-abi.h | 14 +++++++++++++ src/verbs.c | 62 +++++++++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 62 insertions(+), 14 deletions(-) diff --git a/src/mlx5-abi.h b/src/mlx5-abi.h index 769ea811d26b..85f6ee3f691e 100644 --- a/src/mlx5-abi.h +++ b/src/mlx5-abi.h @@ -91,6 +91,20 @@ struct mlx5_create_cq_resp { __u32 cqn; }; +struct mlx5_create_cq_ex { + struct ibv_create_cq_ex ibv_cmd; + __u64 buf_addr; + __u64 db_addr; + __u32 cqe_size; + __u32 comp_mask; +}; + +struct mlx5_create_cq_resp_ex { + struct ibv_create_cq_resp_ex ibv_resp; + __u32 cqn; + __u32 comp_mask; +}; + struct mlx5_create_srq { struct ibv_create_srq ibv_cmd; __u64 buf_addr; diff --git a/src/verbs.c b/src/verbs.c index 1dbee60a6f9a..94a1197351a2 100644 --- a/src/verbs.c +++ b/src/verbs.c @@ -250,17 +250,31 @@ enum { }; enum { - CREATE_CQ_SUPPORTED_FLAGS = IBV_CREATE_CQ_ATTR_COMPLETION_TIMESTAMP + CREATE_CQ_SUPPORTED_FLAGS = IBV_CREATE_CQ_ATTR_COMPLETION_TIMESTAMP | + IBV_CREATE_CQ_ATTR_IGNORE_OVERRUN +}; + +enum cmd_type { + LEGACY_CMD, + EXTENDED_CMD }; static struct ibv_cq *create_cq(struct ibv_context *context, - const struct ibv_create_cq_attr_ex *cq_attr) + struct ibv_create_cq_attr_ex *cq_attr, + enum cmd_type ctype) { struct mlx5_create_cq cmd; + struct mlx5_create_cq_ex cmd_ex; struct mlx5_create_cq_resp resp; + struct mlx5_create_cq_resp_ex resp_ex; struct mlx5_cq *cq; int cqe_sz; - int ret; + /* + * Initialize to non-zero value, because + * ibv_cmd_* should set it to zero in case + * of success execution. + */ + int ret = -1; int ncqe; #ifdef MLX5_DEBUG FILE *fp = to_mctx(context)->dbg_fp; @@ -299,7 +313,6 @@ static struct ibv_cq *create_cq(struct ibv_context *context, return NULL; } - memset(&cmd, 0, sizeof cmd); cq->cons_index = 0; if (mlx5_spinlock_init(&cq->lock)) @@ -342,14 +355,30 @@ static struct ibv_cq *create_cq(struct ibv_context *context, cq->arm_sn = 0; cq->cqe_sz = cqe_sz; - cmd.buf_addr = (uintptr_t) cq->buf_a.buf; - cmd.db_addr = (uintptr_t) cq->dbrec; - cmd.cqe_size = cqe_sz; + if (ctype == LEGACY_CMD) { + memset(&cmd, 0, sizeof(cmd)); + cmd.buf_addr = (uintptr_t) cq->buf_a.buf; + cmd.db_addr = (uintptr_t) cq->dbrec; + cmd.cqe_size = cqe_sz; + + ret = ibv_cmd_create_cq(context, ncqe - 1, cq_attr->channel, + cq_attr->comp_vector, + &cq->ibv_cq, &cmd.ibv_cmd, sizeof cmd, + &resp.ibv_resp, sizeof resp); + } + if (ctype == EXTENDED_CMD) { + memset(&cmd_ex, 0, sizeof(cmd_ex)); + cmd_ex.buf_addr = (uintptr_t) cq->buf_a.buf; + cmd_ex.db_addr = (uintptr_t) cq->dbrec; + cmd_ex.cqe_size = cqe_sz; + + ret = ibv_cmd_create_cq_ex(context, cq_attr, + &cq->ibv_cq, &cmd_ex.ibv_cmd, + sizeof(cmd_ex.ibv_cmd), sizeof(cmd_ex), + &resp_ex.ibv_resp, + sizeof(resp_ex.ibv_resp), sizeof(resp_ex)); + } - ret = ibv_cmd_create_cq(context, ncqe - 1, cq_attr->channel, - cq_attr->comp_vector, - &cq->ibv_cq, &cmd.ibv_cmd, sizeof cmd, - &resp.ibv_resp, sizeof resp); if (ret) { mlx5_dbg(fp, MLX5_DBG_CQ, "ret %d\n", ret); goto err_db; @@ -357,7 +386,12 @@ static struct ibv_cq *create_cq(struct ibv_context *context, cq->active_buf = &cq->buf_a; cq->resize_buf = NULL; - cq->cqn = resp.cqn; + + if (ctype == LEGACY_CMD) + cq->cqn = resp.cqn; + if (ctype == EXTENDED_CMD) + cq->cqn = resp_ex.cqn; + cq->stall_enable = to_mctx(context)->stall_enable; cq->stall_adaptive_enable = to_mctx(context)->stall_adaptive_enable; cq->stall_cycles = to_mctx(context)->stall_cycles; @@ -390,13 +424,13 @@ struct ibv_cq *mlx5_create_cq(struct ibv_context *context, int cqe, .comp_vector = comp_vector, .wc_flags = IBV_WC_STANDARD_FLAGS}; - return create_cq(context, &cq_attr); + return create_cq(context, &cq_attr, LEGACY_CMD); } struct ibv_cq *mlx5_create_cq_ex(struct ibv_context *context, struct ibv_create_cq_attr_ex *cq_attr) { - return create_cq(context, cq_attr); + return create_cq(context, cq_attr, EXTENDED_CMD); } int mlx5_resize_cq(struct ibv_cq *ibcq, int cqe)