From patchwork Mon Aug 19 04:47:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13767908 X-Patchwork-Delegate: jgg@ziepe.ca Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC98314883B for ; Mon, 19 Aug 2024 05:08:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724044129; cv=none; b=g8s2RdzzB3ywHNb+dJa1TmoQ1oJZwyJ12Rp0/tyPaD+DVD2ftyIjRyvsbqQnoDkGzc7U7T6Ca/P164mbLE+caVUwqUYjlji9waKGYdv25+r5dgJBobru9U6nO1svMGdepgq1kd+v058YCykvfvAnkn3twAguoNUO2/VMllS4X34= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724044129; c=relaxed/simple; bh=R4fkHB8Tvr5/l8b4BhtewZUZo4hZLpZ1TRd53xxg15Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=k93TDhaJbD62m6HqG4XRCQzel+Jy+ypG7fNAyKVD4z6bLRrvEK/+Gutj13OJccdo0n5NRLnDVI1mDRBoQQ75AwKPTdkJHo2ojtLUtNIzoB8myPZdmk6K+eHGi6b6hVrOCtU73amvXXkcQVjh3TY3tvZHFYeqB0Vxu7uS10wVsxs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=IJLkYjUN; arc=none smtp.client-ip=209.85.167.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="IJLkYjUN" Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3dd16257b7bso1260587b6e.1 for ; Sun, 18 Aug 2024 22:08:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1724044127; x=1724648927; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=asSOP3AWK79b8AbU9ZiBfPa675PwEVNGQIpGMKJmngI=; b=IJLkYjUN0B0huNSzz1f0spLBBgeHB2NQFldaBtKKd50mQ2cadpQGn9QgANmv30xWxh kp3T8WjJSnm1kBcoM8jR3KUzeYL+RPXsXitoICWRrrblc6g6u6EoAW+yQJ9NQI/wWtVM I+C7ahNaL0cFUHrSP42WSEZaIhrmq3dRsjnaA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724044127; x=1724648927; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=asSOP3AWK79b8AbU9ZiBfPa675PwEVNGQIpGMKJmngI=; b=vwknaVCb6kDXO5aRiyejD8X1w0UvEG4ANkmLmHvFZ/E5QIw++Nd/W5XfVhchvaCNSO NKWvY/w69+MTZl0xTLXbz4JNWFimRQRr7e+Ql8jRoGyRpv58SQ5pUXnMNCNdQ2lJR9Xs rOKBiEzO7egRFvUgAD5uU12471AQkCq9MS551UzEORRkpE6HNPGzd6NGfKwuQGR+hgkD OE4su2KWlLpNLoOYj+2Qqpv+6Oj2+04D7Tol3H8kUrFewcK6o+tVCdJ14Ntv6hSIZRbK KXxSOUribkDNa0dPm0CCOTM0HO7YnA8UayPjBhLRp++uvnlq8vUknhxthTJaCqsiKbog S3VA== X-Gm-Message-State: AOJu0YwLzVk78JohTAhLy/1AlNguaQlj1H/p0JfVlxnDbzEegUtZ1y99 CyUYwtWgi/+90fYvmCKYnvZMW3l22lxHB+534EsucCAhqkI+fViFMQDsGr9NACBOcNl0qYwSoqQ = X-Google-Smtp-Source: AGHT+IFhxB7dBQXTsVnYohe3av3cYxMjm21P6w4bBdck+5VluCc3xZthQOgZfRsrHQDiwn1rA3Q5gg== X-Received: by 2002:a05:6871:b0f:b0:261:1342:26b1 with SMTP id 586e51a60fabf-2701c574f5cmr10544005fac.48.1724044126661; Sun, 18 Aug 2024 22:08:46 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([42.104.124.121]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7c6b61a7672sm6908021a12.4.2024.08.18.22.08.44 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 18 Aug 2024 22:08:45 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Selvin Xavier , Hongguang Gao Subject: [PATCH for-next v3 1/5] RDMA/bnxt_re: Add support for Variable WQE in Genp7 adapters Date: Sun, 18 Aug 2024 21:47:23 -0700 Message-Id: <1724042847-1481-2-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1724042847-1481-1-git-send-email-selvin.xavier@broadcom.com> References: <1724042847-1481-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Variable size WQE means that each send Work Queue Entry to HW can use different WQE sizes as opposed to the static WQE size on the current devices. Set variable WQE mode for Gen P7 devices. Depth of the Queue will be a multiple of slot which is 16 bytes. The number of slots should be a multiple of 256 as per the HW requirement. Initialize the Software shadow queue to hold requests equal to the number of slots. Also, do not expose the variable size WQE capability until the last patch in the series. Signed-off-by: Hongguang Gao Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 8 +++++--- drivers/infiniband/hw/bnxt_re/main.c | 21 +++++++++++---------- drivers/infiniband/hw/bnxt_re/qplib_fp.c | 18 +++++++++--------- drivers/infiniband/hw/bnxt_re/qplib_fp.h | 14 +++++++++++--- drivers/infiniband/hw/bnxt_re/qplib_sp.c | 7 +++++-- drivers/infiniband/hw/bnxt_re/qplib_sp.h | 6 ++++++ 6 files changed, 47 insertions(+), 27 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 7c75735..5073ab1 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -1156,6 +1156,7 @@ static struct bnxt_re_qp *bnxt_re_create_shadow_qp /* Shadow QP SQ depth should be same as QP1 RQ depth */ qp->qplib_qp.sq.wqe_size = bnxt_re_get_wqe_size(0, 6); qp->qplib_qp.sq.max_wqe = qp1_qp->rq.max_wqe; + qp->qplib_qp.sq.max_sw_wqe = qp1_qp->rq.max_wqe; qp->qplib_qp.sq.max_sge = 2; /* Q full delta can be 1 since it is internal QP */ qp->qplib_qp.sq.q_full_delta = 1; @@ -1167,6 +1168,7 @@ static struct bnxt_re_qp *bnxt_re_create_shadow_qp qp->qplib_qp.rq.wqe_size = bnxt_re_get_rwqe_size(6); qp->qplib_qp.rq.max_wqe = qp1_qp->rq.max_wqe; + qp->qplib_qp.rq.max_sw_wqe = qp1_qp->rq.max_wqe; qp->qplib_qp.rq.max_sge = qp1_qp->rq.max_sge; /* Q full delta can be 1 since it is internal QP */ qp->qplib_qp.rq.q_full_delta = 1; @@ -1228,6 +1230,7 @@ static int bnxt_re_init_rq_attr(struct bnxt_re_qp *qp, */ entries = bnxt_re_init_depth(init_attr->cap.max_recv_wr + 1, uctx); rq->max_wqe = min_t(u32, entries, dev_attr->max_qp_wqes + 1); + rq->max_sw_wqe = rq->max_wqe; rq->q_full_delta = 0; rq->sg_info.pgsize = PAGE_SIZE; rq->sg_info.pgshft = PAGE_SHIFT; @@ -1287,6 +1290,7 @@ static int bnxt_re_init_sq_attr(struct bnxt_re_qp *qp, 0 : BNXT_QPLIB_RESERVED_QP_WRS; entries = bnxt_re_init_depth(entries + diff + 1, uctx); sq->max_wqe = min_t(u32, entries, dev_attr->max_qp_wqes + diff + 1); + sq->max_sw_wqe = bnxt_qplib_get_depth(sq, qplqp->wqe_mode, true); sq->q_full_delta = diff + 1; /* * Reserving one slot for Phantom WQE. Application can @@ -2155,6 +2159,7 @@ int bnxt_re_modify_qp(struct ib_qp *ib_qp, struct ib_qp_attr *qp_attr, entries = bnxt_re_init_depth(qp_attr->cap.max_recv_wr, uctx); qp->qplib_qp.rq.max_wqe = min_t(u32, entries, dev_attr->max_qp_wqes + 1); + qp->qplib_qp.rq.max_sw_wqe = qp->qplib_qp.rq.max_wqe; qp->qplib_qp.rq.q_full_delta = qp->qplib_qp.rq.max_wqe - qp_attr->cap.max_recv_wr; qp->qplib_qp.rq.max_sge = qp_attr->cap.max_recv_sge; @@ -4187,9 +4192,6 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata) resp.cqe_sz = sizeof(struct cq_base); resp.max_cqd = dev_attr->max_cq_wqes; - resp.comp_mask |= BNXT_RE_UCNTX_CMASK_HAVE_MODE; - resp.mode = rdev->chip_ctx->modes.wqe_mode; - if (rdev->chip_ctx->modes.db_push) resp.comp_mask |= BNXT_RE_UCNTX_CMASK_WC_DPI_ENABLED; diff --git a/drivers/infiniband/hw/bnxt_re/main.c b/drivers/infiniband/hw/bnxt_re/main.c index 9714b9a..31ba89c 100644 --- a/drivers/infiniband/hw/bnxt_re/main.c +++ b/drivers/infiniband/hw/bnxt_re/main.c @@ -129,13 +129,13 @@ static void bnxt_re_set_db_offset(struct bnxt_re_dev *rdev) } } -static void bnxt_re_set_drv_mode(struct bnxt_re_dev *rdev, u8 mode) +static void bnxt_re_set_drv_mode(struct bnxt_re_dev *rdev) { struct bnxt_qplib_chip_ctx *cctx; cctx = rdev->chip_ctx; - cctx->modes.wqe_mode = bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx) ? - mode : BNXT_QPLIB_WQE_MODE_STATIC; + cctx->modes.wqe_mode = bnxt_qplib_is_chip_gen_p7(rdev->chip_ctx) ? + BNXT_QPLIB_WQE_MODE_VARIABLE : BNXT_QPLIB_WQE_MODE_STATIC; if (bnxt_re_hwrm_qcaps(rdev)) dev_err(rdev_to_dev(rdev), "Failed to query hwrm qcaps\n"); @@ -158,7 +158,7 @@ static void bnxt_re_destroy_chip_ctx(struct bnxt_re_dev *rdev) kfree(chip_ctx); } -static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) +static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev) { struct bnxt_qplib_chip_ctx *chip_ctx; struct bnxt_en_dev *en_dev; @@ -180,7 +180,7 @@ static int bnxt_re_setup_chip_ctx(struct bnxt_re_dev *rdev, u8 wqe_mode) rdev->qplib_res.dattr = &rdev->dev_attr; rdev->qplib_res.is_vf = BNXT_EN_VF(en_dev); - bnxt_re_set_drv_mode(rdev, wqe_mode); + bnxt_re_set_drv_mode(rdev); bnxt_re_set_db_offset(rdev); rc = bnxt_qplib_map_db_bar(&rdev->qplib_res); @@ -1620,7 +1620,7 @@ static void bnxt_re_worker(struct work_struct *work) schedule_delayed_work(&rdev->worker, msecs_to_jiffies(30000)); } -static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) +static int bnxt_re_dev_init(struct bnxt_re_dev *rdev) { struct bnxt_re_ring_attr rattr = {}; struct bnxt_qplib_creq_ctx *creq; @@ -1638,7 +1638,7 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) } set_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags); - rc = bnxt_re_setup_chip_ctx(rdev, wqe_mode); + rc = bnxt_re_setup_chip_ctx(rdev); if (rc) { bnxt_unregister_dev(rdev->en_dev); clear_bit(BNXT_RE_FLAG_NETDEV_REGISTERED, &rdev->flags); @@ -1790,7 +1790,7 @@ static int bnxt_re_dev_init(struct bnxt_re_dev *rdev, u8 wqe_mode) return rc; } -static int bnxt_re_add_device(struct auxiliary_device *adev, u8 wqe_mode) +static int bnxt_re_add_device(struct auxiliary_device *adev) { struct bnxt_aux_priv *aux_priv = container_of(adev, struct bnxt_aux_priv, aux_dev); @@ -1807,7 +1807,7 @@ static int bnxt_re_add_device(struct auxiliary_device *adev, u8 wqe_mode) goto exit; } - rc = bnxt_re_dev_init(rdev, wqe_mode); + rc = bnxt_re_dev_init(rdev); if (rc) goto re_dev_dealloc; @@ -1937,7 +1937,8 @@ static int bnxt_re_probe(struct auxiliary_device *adev, int rc; mutex_lock(&bnxt_re_mutex); - rc = bnxt_re_add_device(adev, BNXT_QPLIB_WQE_MODE_STATIC); + + rc = bnxt_re_add_device(adev); if (rc) { mutex_unlock(&bnxt_re_mutex); return rc; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.c b/drivers/infiniband/hw/bnxt_re/qplib_fp.c index 49e4a4a..0af09e7 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.c @@ -809,13 +809,13 @@ static int bnxt_qplib_alloc_init_swq(struct bnxt_qplib_q *que) { int indx; - que->swq = kcalloc(que->max_wqe, sizeof(*que->swq), GFP_KERNEL); + que->swq = kcalloc(que->max_sw_wqe, sizeof(*que->swq), GFP_KERNEL); if (!que->swq) return -ENOMEM; que->swq_start = 0; - que->swq_last = que->max_wqe - 1; - for (indx = 0; indx < que->max_wqe; indx++) + que->swq_last = que->max_sw_wqe - 1; + for (indx = 0; indx < que->max_sw_wqe; indx++) que->swq[indx].next_idx = indx + 1; que->swq[que->swq_last].next_idx = 0; /* Make it circular */ que->swq_last = 0; @@ -851,7 +851,7 @@ int bnxt_qplib_create_qp1(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) hwq_attr.res = res; hwq_attr.sginfo = &sq->sg_info; hwq_attr.stride = sizeof(struct sq_sge); - hwq_attr.depth = bnxt_qplib_get_depth(sq); + hwq_attr.depth = bnxt_qplib_get_depth(sq, qp->wqe_mode, false); hwq_attr.type = HWQ_TYPE_QUEUE; rc = bnxt_qplib_alloc_init_hwq(&sq->hwq, &hwq_attr); if (rc) @@ -879,7 +879,7 @@ int bnxt_qplib_create_qp1(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) hwq_attr.res = res; hwq_attr.sginfo = &rq->sg_info; hwq_attr.stride = sizeof(struct sq_sge); - hwq_attr.depth = bnxt_qplib_get_depth(rq); + hwq_attr.depth = bnxt_qplib_get_depth(rq, qp->wqe_mode, false); hwq_attr.type = HWQ_TYPE_QUEUE; rc = bnxt_qplib_alloc_init_hwq(&rq->hwq, &hwq_attr); if (rc) @@ -1011,7 +1011,7 @@ int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) hwq_attr.res = res; hwq_attr.sginfo = &sq->sg_info; hwq_attr.stride = sizeof(struct sq_sge); - hwq_attr.depth = bnxt_qplib_get_depth(sq); + hwq_attr.depth = bnxt_qplib_get_depth(sq, qp->wqe_mode, true); hwq_attr.aux_stride = psn_sz; hwq_attr.aux_depth = psn_sz ? bnxt_qplib_set_sq_size(sq, qp->wqe_mode) : 0; @@ -1052,7 +1052,7 @@ int bnxt_qplib_create_qp(struct bnxt_qplib_res *res, struct bnxt_qplib_qp *qp) hwq_attr.res = res; hwq_attr.sginfo = &rq->sg_info; hwq_attr.stride = sizeof(struct sq_sge); - hwq_attr.depth = bnxt_qplib_get_depth(rq); + hwq_attr.depth = bnxt_qplib_get_depth(rq, qp->wqe_mode, false); hwq_attr.aux_stride = 0; hwq_attr.aux_depth = 0; hwq_attr.type = HWQ_TYPE_QUEUE; @@ -2492,7 +2492,7 @@ static int bnxt_qplib_cq_process_req(struct bnxt_qplib_cq *cq, } sq = &qp->sq; - cqe_sq_cons = le16_to_cpu(hwcqe->sq_cons_idx) % sq->max_wqe; + cqe_sq_cons = le16_to_cpu(hwcqe->sq_cons_idx) % sq->max_sw_wqe; if (qp->sq.flushed) { dev_dbg(&cq->hwq.pdev->dev, "%s: QP in Flush QP = %p\n", __func__, qp); @@ -2882,7 +2882,7 @@ static int bnxt_qplib_cq_process_terminal(struct bnxt_qplib_cq *cq, cqe_cons = le16_to_cpu(hwcqe->sq_cons_idx); if (cqe_cons == 0xFFFF) goto do_rq; - cqe_cons %= sq->max_wqe; + cqe_cons %= sq->max_sw_wqe; if (qp->sq.flushed) { dev_dbg(&cq->hwq.pdev->dev, diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h index 56538b9..f54d7a0 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h @@ -251,6 +251,7 @@ struct bnxt_qplib_q { struct bnxt_qplib_db_info dbinfo; struct bnxt_qplib_sg_info sg_info; u32 max_wqe; + u32 max_sw_wqe; u16 wqe_size; u16 q_full_delta; u16 max_sge; @@ -586,15 +587,22 @@ static inline void bnxt_qplib_swq_mod_start(struct bnxt_qplib_q *que, u32 idx) que->swq_start = que->swq[idx].next_idx; } -static inline u32 bnxt_qplib_get_depth(struct bnxt_qplib_q *que) +static inline u32 bnxt_qplib_get_depth(struct bnxt_qplib_q *que, u8 wqe_mode, bool is_sq) { - return (que->wqe_size * que->max_wqe) / sizeof(struct sq_sge); + u32 slots; + + /* Queue depth is the number of slots. */ + slots = (que->wqe_size * que->max_wqe) / sizeof(struct sq_sge); + /* For variable WQE mode, need to align the slots to 256 */ + if (wqe_mode == BNXT_QPLIB_WQE_MODE_VARIABLE && is_sq) + slots = ALIGN(slots, BNXT_VAR_MAX_SLOT_ALIGN); + return slots; } static inline u32 bnxt_qplib_set_sq_size(struct bnxt_qplib_q *que, u8 wqe_mode) { return (wqe_mode == BNXT_QPLIB_WQE_MODE_STATIC) ? - que->max_wqe : bnxt_qplib_get_depth(que); + que->max_wqe : bnxt_qplib_get_depth(que, wqe_mode, true); } static inline u32 bnxt_qplib_set_sq_max_slot(u8 wqe_mode) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.c b/drivers/infiniband/hw/bnxt_re/qplib_sp.c index 9328db9..ca2aa35 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_sp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.c @@ -95,11 +95,13 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw, struct bnxt_qplib_cmdqmsg msg = {}; struct creq_query_func_resp_sb *sb; struct bnxt_qplib_rcfw_sbuf sbuf; + struct bnxt_qplib_chip_ctx *cctx; struct cmdq_query_func req = {}; u8 *tqm_alloc; int i, rc; u32 temp; + cctx = rcfw->res->cctx; bnxt_qplib_rcfw_cmd_prep((struct cmdq_base *)&req, CMDQ_BASE_OPCODE_QUERY_FUNC, sizeof(req)); @@ -133,8 +135,9 @@ int bnxt_qplib_get_dev_attr(struct bnxt_qplib_rcfw *rcfw, * reporting the max number */ attr->max_qp_wqes -= BNXT_QPLIB_RESERVED_QP_WRS + 1; - attr->max_qp_sges = bnxt_qplib_is_chip_gen_p5_p7(rcfw->res->cctx) ? - 6 : sb->max_sge; + + attr->max_qp_sges = cctx->modes.wqe_mode == BNXT_QPLIB_WQE_MODE_VARIABLE ? + min_t(u32, sb->max_sge_var_wqe, BNXT_VAR_MAX_SGE) : 6; attr->max_cq = le32_to_cpu(sb->max_cq); attr->max_cq_wqes = le32_to_cpu(sb->max_cqe); attr->max_cq_sges = attr->max_qp_sges; diff --git a/drivers/infiniband/hw/bnxt_re/qplib_sp.h b/drivers/infiniband/hw/bnxt_re/qplib_sp.h index 16a67d7..a633e2a 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_sp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_sp.h @@ -40,6 +40,7 @@ #ifndef __BNXT_QPLIB_SP_H__ #define __BNXT_QPLIB_SP_H__ +#include #define BNXT_QPLIB_RESERVED_QP_WRS 128 struct bnxt_qplib_dev_attr { @@ -351,4 +352,9 @@ int bnxt_qplib_qext_stat(struct bnxt_qplib_rcfw *rcfw, u32 fid, int bnxt_qplib_modify_cc(struct bnxt_qplib_res *res, struct bnxt_qplib_cc_param *cc_param); +#define BNXT_VAR_MAX_WQE 4352 +#define BNXT_VAR_MAX_SLOT_ALIGN 256 +#define BNXT_VAR_MAX_SGE 13 +#define BNXT_RE_MAX_RQ_WQES 65536 + #endif /* __BNXT_QPLIB_SP_H__*/ From patchwork Mon Aug 19 04:47:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13767909 X-Patchwork-Delegate: jgg@ziepe.ca Received: from mail-il1-f170.google.com (mail-il1-f170.google.com [209.85.166.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 65DC514883B for ; Mon, 19 Aug 2024 05:08:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724044132; cv=none; b=MgtQQsb9UMaqfus0+jbf0Wf4dkFBIDeQ8/orXsRnd5xG12fYWNUeQF0zrwQ9viLaWJbI5IU/4xMBoX6dzA0qXkS8ts+yBCr7GA+3yIVmsZHvOmtL+DPf/qvKnA6uhWkh/htS6Btgnnl+OeozxeJUyVZ6VXLMD30X1ep/VFxPuE0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724044132; c=relaxed/simple; bh=w6IQfTEVVPAsP5lfJO1aCy8NvzAtX6hYoyolL216CZs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=ZP4bldrScDLmOQIM1WgS6RF8V7+v/rcgSsUxZ9saNj4lnXuC/xG/K3XuOeoX1ejVYEyrJgnUEI0sioWNySg0g/6U7NHPi8w+Ha3Xdjz8ZjHBd0Ie3mIz2Za5K07UPN2D/gSG4UODF9ZdG3FDqNoQYBbdnK90u8BP2GU92fSSSXw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=UaaAV5Qe; arc=none smtp.client-ip=209.85.166.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="UaaAV5Qe" Received: by mail-il1-f170.google.com with SMTP id e9e14a558f8ab-39d34be8b64so5626335ab.1 for ; Sun, 18 Aug 2024 22:08:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1724044129; x=1724648929; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=qYN45UtvNHksfr/cCY907kDlv6PSMGdis2SyWN7h9Eg=; b=UaaAV5Qe/r1cgMF3GsJSkLI7TDoUn9pkuQfYd0a6NY1+P50VeJZDhhxAYN1RfubtWW fcKKfKuHd1bgCpt9/RGA17tNYTSUfFeG+qKPpP6bQCkUSw7BmEFJhiG5jgYipB4rrSN5 SIvYwNTcuYuY/kseyziTJh1zRY/8MaBj1ObSw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724044129; x=1724648929; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qYN45UtvNHksfr/cCY907kDlv6PSMGdis2SyWN7h9Eg=; b=lxaFnelY9IczOfeaAS/DPkF46y8l+YYTmTimQJ1wBhYcSDWZPfyTYZyMHpBzbT+0EQ 0AkTaVjXniqLr7e87+4jqbJG3hamRLQu0Mzz+Z4joXK1ommlS/NEqgL11+n1DlKlQz14 VUOKCn9gw8svp1SCEH7LVnkde7Guu5TziAPULk55EEiloUbKfoyJ697wCpMNMms2KIDy wK54iavSnKLu57plqfnwZMYen+BR5BMnLFazwr83GqUXRQFKfAg8hsj3n8oS5ni11ql9 kLumZlWwabnVMTJI1n9n8w4FQGrhcec79YdgIgFJB9BVZysA3ZAP/ekK6NHo9ZTcEn32 vh2Q== X-Gm-Message-State: AOJu0YxTJFXL07rBAt6cC+BoIr7b7fgi+xdec61iGK6umpbvoSGgdMRV PuaShg2wFWvmvy962gk7Vqeb7Yn2+E/e0xIZhlUJpf4lyDenFIdjZ6cUFiPV2g== X-Google-Smtp-Source: AGHT+IGbomdJl2LAuRmX/Ipe7qGY40zEyymTzopR39fJcI8O0avnn8EiGtVyHYxcdR4YHQiSeob1Pw== X-Received: by 2002:a92:cd82:0:b0:39d:1a7d:71ea with SMTP id e9e14a558f8ab-39d26d615fdmr118120035ab.19.1724044129435; Sun, 18 Aug 2024 22:08:49 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([42.104.124.121]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7c6b61a7672sm6908021a12.4.2024.08.18.22.08.47 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 18 Aug 2024 22:08:48 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Selvin Xavier , Hongguang Gao Subject: [PATCH for-next v3 2/5] RDMA/bnxt_re: Get the WQE index from slot index while completing the WQEs Date: Sun, 18 Aug 2024 21:47:24 -0700 Message-Id: <1724042847-1481-3-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1724042847-1481-1-git-send-email-selvin.xavier@broadcom.com> References: <1724042847-1481-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: While reporting the completions, SQ Work Queue index is required to identify the WQE that generated the completions. In variable WQE mode, FW returns the slot index for Error completions. Driver need to walk through the shadow queue between the consumer index and producer index and matches the slot index returned by FW. If a match is found, the next index of the shadow queue is the WQE index to be considered for remaining poll_cq loop. Signed-off-by: Hongguang Gao Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/qplib_fp.c | 43 +++++++++++++++++++++++++++++++- drivers/infiniband/hw/bnxt_re/qplib_fp.h | 10 ++++++++ 2 files changed, 52 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.c b/drivers/infiniband/hw/bnxt_re/qplib_fp.c index 0af09e7..2810ffe 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.c +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.c @@ -2471,6 +2471,32 @@ static int do_wa9060(struct bnxt_qplib_qp *qp, struct bnxt_qplib_cq *cq, return rc; } +static int bnxt_qplib_get_cqe_sq_cons(struct bnxt_qplib_q *sq, u32 cqe_slot) +{ + struct bnxt_qplib_hwq *sq_hwq; + struct bnxt_qplib_swq *swq; + int cqe_sq_cons = -1; + u32 start, last; + + sq_hwq = &sq->hwq; + + start = sq->swq_start; + last = sq->swq_last; + + while (last != start) { + swq = &sq->swq[last]; + if (swq->slot_idx == cqe_slot) { + cqe_sq_cons = swq->next_idx; + dev_err(&sq_hwq->pdev->dev, "%s: Found cons wqe = %d slot = %d\n", + __func__, cqe_sq_cons, cqe_slot); + break; + } + + last = swq->next_idx; + } + return cqe_sq_cons; +} + static int bnxt_qplib_cq_process_req(struct bnxt_qplib_cq *cq, struct cq_req *hwcqe, struct bnxt_qplib_cqe **pcqe, int *budget, @@ -2478,9 +2504,10 @@ static int bnxt_qplib_cq_process_req(struct bnxt_qplib_cq *cq, { struct bnxt_qplib_swq *swq; struct bnxt_qplib_cqe *cqe; + u32 cqe_sq_cons, slot_num; struct bnxt_qplib_qp *qp; struct bnxt_qplib_q *sq; - u32 cqe_sq_cons; + int cqe_cons; int rc = 0; qp = (struct bnxt_qplib_qp *)((unsigned long) @@ -2498,6 +2525,20 @@ static int bnxt_qplib_cq_process_req(struct bnxt_qplib_cq *cq, "%s: QP in Flush QP = %p\n", __func__, qp); goto done; } + + if (__is_err_cqe_for_var_wqe(qp, hwcqe->status)) { + slot_num = le16_to_cpu(hwcqe->sq_cons_idx); + cqe_cons = bnxt_qplib_get_cqe_sq_cons(sq, slot_num); + if (cqe_cons < 0) { + dev_err(&cq->hwq.pdev->dev, "%s: Wrong SQ cons cqe_slot_indx = %d\n", + __func__, slot_num); + goto done; + } + cqe_sq_cons = cqe_cons; + dev_err(&cq->hwq.pdev->dev, "%s: cqe_sq_cons = %d swq_last = %d swq_start = %d\n", + __func__, cqe_sq_cons, sq->swq_last, sq->swq_start); + } + /* Require to walk the sq's swq to fabricate CQEs for all previously * signaled SWQEs due to CQE aggregation from the current sq cons * to the cqe_sq_cons diff --git a/drivers/infiniband/hw/bnxt_re/qplib_fp.h b/drivers/infiniband/hw/bnxt_re/qplib_fp.h index f54d7a0..2e7a4fd 100644 --- a/drivers/infiniband/hw/bnxt_re/qplib_fp.h +++ b/drivers/infiniband/hw/bnxt_re/qplib_fp.h @@ -649,4 +649,14 @@ static inline __le64 bnxt_re_update_msn_tbl(u32 st_idx, u32 npsn, u32 start_psn) (((start_psn) << SQ_MSN_SEARCH_START_PSN_SFT) & SQ_MSN_SEARCH_START_PSN_MASK)); } + +static inline bool __is_var_wqe(struct bnxt_qplib_qp *qp) +{ + return (qp->wqe_mode == BNXT_QPLIB_WQE_MODE_VARIABLE); +} + +static inline bool __is_err_cqe_for_var_wqe(struct bnxt_qplib_qp *qp, u8 status) +{ + return (status != CQ_REQ_STATUS_OK) && __is_var_wqe(qp); +} #endif /* __BNXT_QPLIB_FP_H__ */ From patchwork Mon Aug 19 04:47:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13767910 X-Patchwork-Delegate: jgg@ziepe.ca Received: from mail-ot1-f53.google.com (mail-ot1-f53.google.com [209.85.210.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D2621474CE for ; Mon, 19 Aug 2024 05:08:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724044134; cv=none; b=PQLFG9zpe7qYHs6KLaCou8BPg9wP9mp0UsC09VZuwNTlrAv1+wAr4S0X13DsdRk7xyHE7zcPWgnmOlbVoSIzN6pGnEMhXIqazrG8I97hh9gkD6HOUS70gAhokfe5bqBFo1PZ/ZnPcWIfyBNEgWGEho3I/3ng0zKIg5+ytn5LSjs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724044134; c=relaxed/simple; bh=1/geaDc1BsrJyyrN7G8SbKsMVujIayqQAPYfTTyIitw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=LJskLISGwhKPln+pKVBVeuV3spkJWBeD7HiWapnwY0bx6cprLR1MjYE2P+6+dpsmkcjziNijxgnyUc7su7+a98rMUFMiftYg1BB0QsGlO1VXQFoWeA+JrLB02dH2YTNNaBVxClkoctRIoYfTmYRyk4ilD/WJZy6MSAoeUnyhLoE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=TdTunkS6; arc=none smtp.client-ip=209.85.210.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="TdTunkS6" Received: by mail-ot1-f53.google.com with SMTP id 46e09a7af769-7093f3a1af9so2453032a34.1 for ; Sun, 18 Aug 2024 22:08:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1724044132; x=1724648932; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=d2DSdWyXag+IfyFGtcfwPMLPYZ6uqtC0MqWW+pkBibs=; b=TdTunkS6XSkrfHKJMYnz1L1SvHIdi31Fp9s+Wxo6DllJG7/A0vd/uJnS5ui5VZZa4s tpabRwJHjceCw4fRwB/Ko2WO0ET5zHp4Ul87wWkO4FSWjy+AZr6ArHR6q9V+4LFLoWgV bLKalGN4GW/YphD9fXWjYodd0Xb6hc4ur6p9g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724044132; x=1724648932; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=d2DSdWyXag+IfyFGtcfwPMLPYZ6uqtC0MqWW+pkBibs=; b=oYb8ysiol+eJy2DI45zaTA8LsWSc9IUkrHyEapxML5cvD21wnK/er6tcZC0NZx1PCv UBvTNy5/EXrDeZGcL1Oqno6c9gBJi4qXsC7b3T9smsAINkP8Sr/6DDjjW2lPBAuThzgh X2zU2FRq2ZV5xFIVi23bYjI3UKQQZznwHLIBtpXwx1nj3BapZIwYe7Y0rU8euDIgT3d3 g8BBVYapbZt5wi6XX+eFXW4YZXQv6qe1xSmwM2U0Slzm62Ta3fVWoWcTJxCj3QgjKz42 2mI/JTzMZ65BBb+gSIc6Rly41JZUW4SaJ7+gHID9XNbJLaX8Rz+g9pf8MjvYy1nM247h vcdA== X-Gm-Message-State: AOJu0Yx1r1YV7E3OZrwRauZS9iPg2iGY+M9qP3ECDb2wycf1i2Tc/rkO pgMSOWbY3MEZ7U0huIlM1wi0gQrEODVPV5ZgXuPYRaTPNLsFxMCD/f6fjtx8ea44V+684DVFqP8 = X-Google-Smtp-Source: AGHT+IGaQz9OEVc6gX4SZbmI8hPq7aRFVtY90MqRgS758lo0XlTkYUVlKNMN6+oFlUV4TDPsuJkM+Q== X-Received: by 2002:a05:6830:3111:b0:709:50e6:489d with SMTP id 46e09a7af769-70cac83be49mr11472873a34.3.1724044132033; Sun, 18 Aug 2024 22:08:52 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([42.104.124.121]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7c6b61a7672sm6908021a12.4.2024.08.18.22.08.49 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 18 Aug 2024 22:08:51 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Selvin Xavier Subject: [PATCH for-next v3 3/5] RDMA/bnxt_re: Fix the table size for PSN/MSN entries Date: Sun, 18 Aug 2024 21:47:25 -0700 Message-Id: <1724042847-1481-4-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1724042847-1481-1-git-send-email-selvin.xavier@broadcom.com> References: <1724042847-1481-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: HW MSN table size is always a power of 2. So the pages should be mapped accordingly. Use the power of two calculation while get the number of PSN/MSN entries. Fixes: 6f6bfbc595fb ("RDMA/bnxt_re: Expose the MSN table capability for user library") Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 5073ab1..4dd137b 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -1042,6 +1042,8 @@ static int bnxt_re_init_user_qp(struct bnxt_re_dev *rdev, struct bnxt_re_pd *pd, qplib_qp->sq.max_wqe : ((qplib_qp->sq.max_wqe * qplib_qp->sq.wqe_size) / sizeof(struct bnxt_qplib_sge)); + if (_is_host_msn_table(rdev->qplib_res.dattr->dev_cap_flags2)) + psn_nume = roundup_pow_of_two(psn_nume); bytes += (psn_nume * psn_sz); } From patchwork Mon Aug 19 04:47:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13767911 X-Patchwork-Delegate: jgg@ziepe.ca Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC7C314A0B9 for ; Mon, 19 Aug 2024 05:08:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724044137; cv=none; b=cmrlUPvAKsONAaQmTBPrWcnE7WUK6wtRuKcMFdkK9dvqFV3t6Wng8w8wKPl0NlSWZePEwNcFsAW0boxp3o6DGx3G9sVnU3K2LtwNytSPb05T7vLTNKngRv0YIPb6UT36Ps0Vr2y5dK1+6az+P9WvkmucbpKDFf7Y0KgEC1BLfm8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724044137; c=relaxed/simple; bh=v7nZDKAqLK2Ey0TRKv+HyCljEK+XIgACJYBjHj3gx3M=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=IjhQXd9xxlJNMqQiclvZ6DYBqNqjmzXAmfYXCvji/u/oRTLccgsWZ4bfa0e7ce9UGGnWBcAGku2RBHZoVSahg3HjZXn7G69GZ8PK0iJ6F+hzb/hd/in1wJsY2Rkd+KXb8lxDr0Hbfn26bvhuE3JO6rPLxVM0UkxmWfKOMqsktZw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=X/xbyOLt; arc=none smtp.client-ip=209.85.210.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="X/xbyOLt" Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-710d0995e21so2831002b3a.1 for ; Sun, 18 Aug 2024 22:08:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1724044135; x=1724648935; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=An7ht5yhz9XZPhxRnRN9qqRfbMXY428fZ5j0ExIEwQs=; b=X/xbyOLtzR2hjtBFaaDL8WeqKhokUtEXARbJWAWuPrrVpZEP0RpBMOBis3wk2nI537 s4ZkK5kX/6/lzZkcGtRRANMeVVb6L7CcN+87oilb/wxp+5rtlUsdtHAyUmklofQ0aojy MlGgUJqNbvDkySiWeZrAoG/tLCw/xKYCFYCSc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724044135; x=1724648935; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=An7ht5yhz9XZPhxRnRN9qqRfbMXY428fZ5j0ExIEwQs=; b=EBJ8JvRVAXjRWQS4jER83Wo4OEHhj3QxZE4I3uDFAK57w/Y7Yme0SgLM3vdi1NSSuu hSHb74Zy9NxQAtrOjZQPR/LW+tR2kGWQvXgE828zfY/qnTItaticXqCuB/hcnwhTzFgq 0T0WahaWQmcno8AJGj7buY/T9+ZgX+EnKGTD/NuO6bnMjxmUcmnRE3daDOlZAeX5fEEC 73LZbWgDVJWtC0xgco+q5lRm0ciUhcaPW7UsK/uEe4AuuQldl0J8onxOF15rAQcd3GmX bKHPxQnRo9snMwMv8GyX9K8OY8ypfTxC0bT2F8rxQ9DZM47Flm+ZEEX0RIl8ADYoDMlL yBsw== X-Gm-Message-State: AOJu0YwBk/Is3TPKZl7X8JZwVblGu00BD1uaUV5jgHKk5ySMCTjmayQJ 6z3Oc9/OJZHEQuFFdLuy6qYpTnvpQgTsPEFtIzCUXgmvdkiH0TySRvYrGMF3LQ== X-Google-Smtp-Source: AGHT+IF/N+KhWQLtI9x+5u7Fm1w4adznR3VTrcoa8DQl54rk1blw+zS0TC8r5XF8RceGmo7xbRfY/A== X-Received: by 2002:a05:6a00:1acf:b0:712:7414:8837 with SMTP id d2e1a72fcca58-713c4e3e116mr9368568b3a.12.1724044134832; Sun, 18 Aug 2024 22:08:54 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([42.104.124.121]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7c6b61a7672sm6908021a12.4.2024.08.18.22.08.52 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 18 Aug 2024 22:08:54 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Selvin Xavier , Hongguang Gao Subject: [PATCH for-next v3 4/5] RDMA/bnxt_re: Handle variable WQE support for user applications Date: Sun, 18 Aug 2024 21:47:26 -0700 Message-Id: <1724042847-1481-5-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1724042847-1481-1-git-send-email-selvin.xavier@broadcom.com> References: <1724042847-1481-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: User library calculates the number of slots required for user applications and it can pass that information to the driver. Driver can use this value and update the HW directly. This mechanism is currently used only for the newly introduced variable size WQEs. Extend the bnxt_re_qp_req structure to pass the Send Queue slot count. Reorganize the code to get the sq_slots before initializing the Send Queue attributes. Signed-off-by: Hongguang Gao Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 106 ++++++++++++++++++------------- drivers/infiniband/hw/bnxt_re/ib_verbs.h | 16 ++++- include/uapi/rdma/bnxt_re-abi.h | 6 ++ 3 files changed, 82 insertions(+), 46 deletions(-) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 4dd137b..2932db1 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -1017,20 +1017,15 @@ static int bnxt_re_setup_swqe_size(struct bnxt_re_qp *qp, } static int bnxt_re_init_user_qp(struct bnxt_re_dev *rdev, struct bnxt_re_pd *pd, - struct bnxt_re_qp *qp, struct ib_udata *udata) + struct bnxt_re_qp *qp, struct bnxt_re_ucontext *cntx, + struct bnxt_re_qp_req *ureq) { struct bnxt_qplib_qp *qplib_qp; - struct bnxt_re_ucontext *cntx; - struct bnxt_re_qp_req ureq; int bytes = 0, psn_sz; struct ib_umem *umem; int psn_nume; qplib_qp = &qp->qplib_qp; - cntx = rdma_udata_to_drv_context(udata, struct bnxt_re_ucontext, - ib_uctx); - if (ib_copy_from_udata(&ureq, udata, sizeof(ureq))) - return -EFAULT; bytes = (qplib_qp->sq.max_wqe * qplib_qp->sq.wqe_size); /* Consider mapping PSN search memory only for RC QPs. */ @@ -1038,17 +1033,20 @@ static int bnxt_re_init_user_qp(struct bnxt_re_dev *rdev, struct bnxt_re_pd *pd, psn_sz = bnxt_qplib_is_chip_gen_p5_p7(rdev->chip_ctx) ? sizeof(struct sq_psn_search_ext) : sizeof(struct sq_psn_search); - psn_nume = (qplib_qp->wqe_mode == BNXT_QPLIB_WQE_MODE_STATIC) ? - qplib_qp->sq.max_wqe : - ((qplib_qp->sq.max_wqe * qplib_qp->sq.wqe_size) / - sizeof(struct bnxt_qplib_sge)); + if (cntx && bnxt_re_is_var_size_supported(rdev, cntx)) { + psn_nume = ureq->sq_slots; + } else { + psn_nume = (qplib_qp->wqe_mode == BNXT_QPLIB_WQE_MODE_STATIC) ? + qplib_qp->sq.max_wqe : ((qplib_qp->sq.max_wqe * qplib_qp->sq.wqe_size) / + sizeof(struct bnxt_qplib_sge)); + } if (_is_host_msn_table(rdev->qplib_res.dattr->dev_cap_flags2)) psn_nume = roundup_pow_of_two(psn_nume); bytes += (psn_nume * psn_sz); } bytes = PAGE_ALIGN(bytes); - umem = ib_umem_get(&rdev->ibdev, ureq.qpsva, bytes, + umem = ib_umem_get(&rdev->ibdev, ureq->qpsva, bytes, IB_ACCESS_LOCAL_WRITE); if (IS_ERR(umem)) return PTR_ERR(umem); @@ -1057,12 +1055,12 @@ static int bnxt_re_init_user_qp(struct bnxt_re_dev *rdev, struct bnxt_re_pd *pd, qplib_qp->sq.sg_info.umem = umem; qplib_qp->sq.sg_info.pgsize = PAGE_SIZE; qplib_qp->sq.sg_info.pgshft = PAGE_SHIFT; - qplib_qp->qp_handle = ureq.qp_handle; + qplib_qp->qp_handle = ureq->qp_handle; if (!qp->qplib_qp.srq) { bytes = (qplib_qp->rq.max_wqe * qplib_qp->rq.wqe_size); bytes = PAGE_ALIGN(bytes); - umem = ib_umem_get(&rdev->ibdev, ureq.qprva, bytes, + umem = ib_umem_get(&rdev->ibdev, ureq->qprva, bytes, IB_ACCESS_LOCAL_WRITE); if (IS_ERR(umem)) goto rqfail; @@ -1261,14 +1259,15 @@ static void bnxt_re_adjust_gsi_rq_attr(struct bnxt_re_qp *qp) static int bnxt_re_init_sq_attr(struct bnxt_re_qp *qp, struct ib_qp_init_attr *init_attr, - struct bnxt_re_ucontext *uctx) + struct bnxt_re_ucontext *uctx, + struct bnxt_re_qp_req *ureq) { struct bnxt_qplib_dev_attr *dev_attr; struct bnxt_qplib_qp *qplqp; struct bnxt_re_dev *rdev; struct bnxt_qplib_q *sq; + int diff = 0; int entries; - int diff; int rc; rdev = qp->rdev; @@ -1277,22 +1276,28 @@ static int bnxt_re_init_sq_attr(struct bnxt_re_qp *qp, dev_attr = &rdev->dev_attr; sq->max_sge = init_attr->cap.max_send_sge; - if (sq->max_sge > dev_attr->max_qp_sges) { - sq->max_sge = dev_attr->max_qp_sges; - init_attr->cap.max_send_sge = sq->max_sge; - } + entries = init_attr->cap.max_send_wr; + if (uctx && qplqp->wqe_mode == BNXT_QPLIB_WQE_MODE_VARIABLE) { + sq->max_wqe = ureq->sq_slots; + sq->max_sw_wqe = ureq->sq_slots; + sq->wqe_size = sizeof(struct sq_sge); + } else { + if (sq->max_sge > dev_attr->max_qp_sges) { + sq->max_sge = dev_attr->max_qp_sges; + init_attr->cap.max_send_sge = sq->max_sge; + } - rc = bnxt_re_setup_swqe_size(qp, init_attr); - if (rc) - return rc; + rc = bnxt_re_setup_swqe_size(qp, init_attr); + if (rc) + return rc; - entries = init_attr->cap.max_send_wr; - /* Allocate 128 + 1 more than what's provided */ - diff = (qplqp->wqe_mode == BNXT_QPLIB_WQE_MODE_VARIABLE) ? - 0 : BNXT_QPLIB_RESERVED_QP_WRS; - entries = bnxt_re_init_depth(entries + diff + 1, uctx); - sq->max_wqe = min_t(u32, entries, dev_attr->max_qp_wqes + diff + 1); - sq->max_sw_wqe = bnxt_qplib_get_depth(sq, qplqp->wqe_mode, true); + /* Allocate 128 + 1 more than what's provided */ + diff = (qplqp->wqe_mode == BNXT_QPLIB_WQE_MODE_VARIABLE) ? + 0 : BNXT_QPLIB_RESERVED_QP_WRS; + entries = bnxt_re_init_depth(entries + diff + 1, uctx); + sq->max_wqe = min_t(u32, entries, dev_attr->max_qp_wqes + diff + 1); + sq->max_sw_wqe = bnxt_qplib_get_depth(sq, qplqp->wqe_mode, true); + } sq->q_full_delta = diff + 1; /* * Reserving one slot for Phantom WQE. Application can @@ -1355,10 +1360,10 @@ static int bnxt_re_init_qp_type(struct bnxt_re_dev *rdev, static int bnxt_re_init_qp_attr(struct bnxt_re_qp *qp, struct bnxt_re_pd *pd, struct ib_qp_init_attr *init_attr, - struct ib_udata *udata) + struct bnxt_re_ucontext *uctx, + struct bnxt_re_qp_req *ureq) { struct bnxt_qplib_dev_attr *dev_attr; - struct bnxt_re_ucontext *uctx; struct bnxt_qplib_qp *qplqp; struct bnxt_re_dev *rdev; struct bnxt_re_cq *cq; @@ -1368,7 +1373,6 @@ static int bnxt_re_init_qp_attr(struct bnxt_re_qp *qp, struct bnxt_re_pd *pd, qplqp = &qp->qplib_qp; dev_attr = &rdev->dev_attr; - uctx = rdma_udata_to_drv_context(udata, struct bnxt_re_ucontext, ib_uctx); /* Setup misc params */ ether_addr_copy(qplqp->smac, rdev->netdev->dev_addr); qplqp->pd = &pd->qplib_pd; @@ -1381,8 +1385,7 @@ static int bnxt_re_init_qp_attr(struct bnxt_re_qp *qp, struct bnxt_re_pd *pd, goto out; } qplqp->type = (u8)qptype; - qplqp->wqe_mode = rdev->chip_ctx->modes.wqe_mode; - + qplqp->wqe_mode = bnxt_re_is_var_size_supported(rdev, uctx); if (init_attr->qp_type == IB_QPT_RC) { qplqp->max_rd_atomic = dev_attr->max_qp_rd_atom; qplqp->max_dest_rd_atomic = dev_attr->max_qp_init_rd_atom; @@ -1417,14 +1420,14 @@ static int bnxt_re_init_qp_attr(struct bnxt_re_qp *qp, struct bnxt_re_pd *pd, bnxt_re_adjust_gsi_rq_attr(qp); /* Setup SQ */ - rc = bnxt_re_init_sq_attr(qp, init_attr, uctx); + rc = bnxt_re_init_sq_attr(qp, init_attr, uctx, ureq); if (rc) goto out; if (init_attr->qp_type == IB_QPT_GSI) bnxt_re_adjust_gsi_sq_attr(qp, init_attr, uctx); - if (udata) /* This will update DPI and qp_handle */ - rc = bnxt_re_init_user_qp(rdev, pd, qp, udata); + if (uctx) /* This will update DPI and qp_handle */ + rc = bnxt_re_init_user_qp(rdev, pd, qp, uctx, ureq); out: return rc; } @@ -1525,14 +1528,27 @@ static bool bnxt_re_test_qp_limits(struct bnxt_re_dev *rdev, int bnxt_re_create_qp(struct ib_qp *ib_qp, struct ib_qp_init_attr *qp_init_attr, struct ib_udata *udata) { - struct ib_pd *ib_pd = ib_qp->pd; - struct bnxt_re_pd *pd = container_of(ib_pd, struct bnxt_re_pd, ib_pd); - struct bnxt_re_dev *rdev = pd->rdev; - struct bnxt_qplib_dev_attr *dev_attr = &rdev->dev_attr; - struct bnxt_re_qp *qp = container_of(ib_qp, struct bnxt_re_qp, ib_qp); + struct bnxt_qplib_dev_attr *dev_attr; + struct bnxt_re_ucontext *uctx; + struct bnxt_re_qp_req ureq; + struct bnxt_re_dev *rdev; + struct bnxt_re_pd *pd; + struct bnxt_re_qp *qp; + struct ib_pd *ib_pd; u32 active_qps; int rc; + ib_pd = ib_qp->pd; + pd = container_of(ib_pd, struct bnxt_re_pd, ib_pd); + rdev = pd->rdev; + dev_attr = &rdev->dev_attr; + qp = container_of(ib_qp, struct bnxt_re_qp, ib_qp); + + uctx = rdma_udata_to_drv_context(udata, struct bnxt_re_ucontext, ib_uctx); + if (udata) + if (ib_copy_from_udata(&ureq, udata, min(udata->inlen, sizeof(ureq)))) + return -EFAULT; + rc = bnxt_re_test_qp_limits(rdev, qp_init_attr, dev_attr); if (!rc) { rc = -EINVAL; @@ -1540,7 +1556,7 @@ int bnxt_re_create_qp(struct ib_qp *ib_qp, struct ib_qp_init_attr *qp_init_attr, } qp->rdev = rdev; - rc = bnxt_re_init_qp_attr(qp, pd, qp_init_attr, udata); + rc = bnxt_re_init_qp_attr(qp, pd, qp_init_attr, uctx, &ureq); if (rc) goto fail; @@ -4215,7 +4231,7 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata) goto cfail; if (ureq.comp_mask & BNXT_RE_COMP_MASK_REQ_UCNTX_POW2_SUPPORT) { resp.comp_mask |= BNXT_RE_UCNTX_CMASK_POW2_DISABLED; - uctx->cmask |= BNXT_RE_UCNTX_CMASK_POW2_DISABLED; + uctx->cmask |= BNXT_RE_UCNTX_CAP_POW2_DISABLED; } } diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.h b/drivers/infiniband/hw/bnxt_re/ib_verbs.h index e98cb17..7c8350f 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.h +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.h @@ -171,12 +171,26 @@ static inline u16 bnxt_re_get_rwqe_size(int nsge) return sizeof(struct rq_wqe_hdr) + (nsge * sizeof(struct sq_sge)); } +enum { + BNXT_RE_UCNTX_CAP_POW2_DISABLED = 0x1ULL, + BNXT_RE_UCNTX_CAP_VAR_WQE_ENABLED = 0x2ULL, +}; + static inline u32 bnxt_re_init_depth(u32 ent, struct bnxt_re_ucontext *uctx) { - return uctx ? (uctx->cmask & BNXT_RE_UCNTX_CMASK_POW2_DISABLED) ? + return uctx ? (uctx->cmask & BNXT_RE_UCNTX_CAP_POW2_DISABLED) ? ent : roundup_pow_of_two(ent) : ent; } +static inline bool bnxt_re_is_var_size_supported(struct bnxt_re_dev *rdev, + struct bnxt_re_ucontext *uctx) +{ + if (uctx) + return uctx->cmask & BNXT_RE_UCNTX_CAP_VAR_WQE_ENABLED; + else + return rdev->chip_ctx->modes.wqe_mode; +} + int bnxt_re_query_device(struct ib_device *ibdev, struct ib_device_attr *ib_attr, struct ib_udata *udata); diff --git a/include/uapi/rdma/bnxt_re-abi.h b/include/uapi/rdma/bnxt_re-abi.h index e61104f..7114061 100644 --- a/include/uapi/rdma/bnxt_re-abi.h +++ b/include/uapi/rdma/bnxt_re-abi.h @@ -118,10 +118,16 @@ struct bnxt_re_resize_cq_req { __aligned_u64 cq_va; }; +enum bnxt_re_qp_mask { + BNXT_RE_QP_REQ_MASK_VAR_WQE_SQ_SLOTS = 0x1, +}; + struct bnxt_re_qp_req { __aligned_u64 qpsva; __aligned_u64 qprva; __aligned_u64 qp_handle; + __aligned_u64 comp_mask; + __u32 sq_slots; }; struct bnxt_re_qp_resp { From patchwork Mon Aug 19 04:47:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Selvin Xavier X-Patchwork-Id: 13767912 X-Patchwork-Delegate: jgg@ziepe.ca Received: from mail-ot1-f42.google.com (mail-ot1-f42.google.com [209.85.210.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 828C714A0B9 for ; Mon, 19 Aug 2024 05:08:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724044140; cv=none; b=WUtp7Ohxk9Dy7HLP8x10AV9bntZAe4MX4E7o3pmunFrfP/1u8YC4vtp5KqgpMoCChk8CNb+D1p/4CuqoPv+P0R+8I7MB0ft5JjI9yXzOnGkHX2sw+pUmGCWEDrQPmMsypuCE9yyH45BKKXPn1opY64zca/hywPnRcKBmMHuL0IM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724044140; c=relaxed/simple; bh=TahX3B1Iuw1jEr2+u0UIDJab1OxyO54LI0+bMBFXKUU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References; b=JvWl+3hmTaIP2WAMgHiFLvb/O546qNSsOvTYzJBuPiJiexnV4rSWST3QffQfbIEJ4RstscLJBRaUAR2CCNyZ8mkS+TCF6JJNG4UFlzWp/18975+Re/KZEiQWRmVBRiOI0RnLLIWkpFOWqBbz6fNC73hLJlqoCdy90ZXKfMC2FBo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=Uy7Evz5K; arc=none smtp.client-ip=209.85.210.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="Uy7Evz5K" Received: by mail-ot1-f42.google.com with SMTP id 46e09a7af769-709346604a7so1882237a34.1 for ; Sun, 18 Aug 2024 22:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1724044137; x=1724648937; darn=vger.kernel.org; h=references:in-reply-to:message-id:date:subject:cc:to:from:from:to :cc:subject:date:message-id:reply-to; bh=+leSXdUbIYKI7LvY2C7MNiQQxTay47JkFU/iIQbOOFI=; b=Uy7Evz5KJ6PSXaFFZL4fEeyicnDdqtmYor9+GIP8hV3WY8QbET51DEs++0nKxlrt5r P/6OqpAU7NQtD8D8Lf+Nw6xZ/qwh1YtDlAylUtiq4rXf0V9YZXxRCd4yovBEuVbHMhtX B9pPhauqwFnLrG4XWsTyM6U8PimlJJtlnEPwk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724044137; x=1724648937; h=references:in-reply-to:message-id:date:subject:cc:to:from :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+leSXdUbIYKI7LvY2C7MNiQQxTay47JkFU/iIQbOOFI=; b=npDLCLWEF6jc7YaZbLm9FaYrjh1y+5veLkKti74q4KvgqG6kmxc1VydhKY8311+EwM i34HtJbbG16spmyhbrym0bEYBa6P85N3XXhs5jYNNmcDt8HRDxmHcSaCFelk8ulAMInE kWHQYSkxROdqSeS8IgqaiSm5Dm9zPDyGdXvHOT3HcsSyyiwTWvTw/8anO2Ha+m4QYitO vbJ9OE3iHqUS7yZD/zt2t/R5V38zzsfCvduT7JRl6jAuEDQlQ9mt5WQOnFjtuEw7SyS9 ttp2eLv+UfidZtDINUNITy5HGDByWbfn7z6c6F3r3JmYbU3nsU44vux86Gg8PlQKYC4r IlWQ== X-Gm-Message-State: AOJu0YyXwTO/IdSMAjnN1eBzYnkutEbMyY5dPNebXmTMeZfwY2AuyWcP ECQX5Ly82m86KXb4frTn0jUazzZ2GeR6hD/qOXg788dg+Y4kkqUVIh1feHyQPg== X-Google-Smtp-Source: AGHT+IFZG/Vxpi/Bxi0mll2sYNYAFKuYkASE99SXga3KHUUeMjmMePxK0TmDf9NHsrABjlYqilwNkg== X-Received: by 2002:a05:6830:d8b:b0:708:72d7:223f with SMTP id 46e09a7af769-70cac8c1cb3mr14486198a34.22.1724044137595; Sun, 18 Aug 2024 22:08:57 -0700 (PDT) Received: from sxavier-dev.dhcp.broadcom.net ([42.104.124.121]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7c6b61a7672sm6908021a12.4.2024.08.18.22.08.55 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 18 Aug 2024 22:08:57 -0700 (PDT) From: Selvin Xavier To: leon@kernel.org, jgg@ziepe.ca Cc: linux-rdma@vger.kernel.org, andrew.gospodarek@broadcom.com, Selvin Xavier , Hongguang Gao Subject: [PATCH for-next v3 5/5] RDMA/bnxt_re: Enable variable size WQEs for user space applications Date: Sun, 18 Aug 2024 21:47:27 -0700 Message-Id: <1724042847-1481-6-git-send-email-selvin.xavier@broadcom.com> X-Mailer: git-send-email 2.5.5 In-Reply-To: <1724042847-1481-1-git-send-email-selvin.xavier@broadcom.com> References: <1724042847-1481-1-git-send-email-selvin.xavier@broadcom.com> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Add backward compatibility code to enable variable size WQEs only if the user lib supports it. Signed-off-by: Hongguang Gao Signed-off-by: Selvin Xavier --- drivers/infiniband/hw/bnxt_re/ib_verbs.c | 5 +++++ include/uapi/rdma/bnxt_re-abi.h | 1 + 2 files changed, 6 insertions(+) diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 2932db1..82444fd 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -4233,6 +4233,11 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata) resp.comp_mask |= BNXT_RE_UCNTX_CMASK_POW2_DISABLED; uctx->cmask |= BNXT_RE_UCNTX_CAP_POW2_DISABLED; } + if (ureq.comp_mask & BNXT_RE_COMP_MASK_REQ_UCNTX_VAR_WQE_SUPPORT) { + resp.comp_mask |= BNXT_RE_UCNTX_CMASK_HAVE_MODE; + resp.mode = rdev->chip_ctx->modes.wqe_mode; + uctx->cmask |= BNXT_RE_UCNTX_CAP_VAR_WQE_ENABLED; + } } rc = ib_copy_to_udata(udata, &resp, min(udata->outlen, sizeof(resp))); diff --git a/include/uapi/rdma/bnxt_re-abi.h b/include/uapi/rdma/bnxt_re-abi.h index 7114061..6821002 100644 --- a/include/uapi/rdma/bnxt_re-abi.h +++ b/include/uapi/rdma/bnxt_re-abi.h @@ -66,6 +66,7 @@ enum bnxt_re_wqe_mode { enum { BNXT_RE_COMP_MASK_REQ_UCNTX_POW2_SUPPORT = 0x01, + BNXT_RE_COMP_MASK_REQ_UCNTX_VAR_WQE_SUPPORT = 0x02, }; struct bnxt_re_uctx_req {