From patchwork Tue Jun 15 18:31:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 12322645 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 494A2C48BE8 for ; Tue, 15 Jun 2021 18:32:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 22A38610F7 for ; Tue, 15 Jun 2021 18:32:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231153AbhFOSeH (ORCPT ); Tue, 15 Jun 2021 14:34:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230374AbhFOSeH (ORCPT ); Tue, 15 Jun 2021 14:34:07 -0400 Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 73F3DC061574 for ; Tue, 15 Jun 2021 11:32:02 -0700 (PDT) Received: by mail-pj1-x102c.google.com with SMTP id g6-20020a17090adac6b029015d1a9a6f1aso2241045pjx.1 for ; Tue, 15 Jun 2021 11:32:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=Z9CKkjG3vxVnDuVHBP4vYmsiDKebLgcksrfYYlPK73o=; b=Cw8Xxxrjbw1ya4o3hU6hoW0s+lnjeE6omGPiDsv62hnOYLL33SyH/CEPsVQMD2cAgU nDsXqZ+82Rj1+VQ7eaJqx15tBO7BTAr7VKmXX4DpM+YMBikcXhwV9/DT/9vBznK3a+de HWCpQUMlfNEzvqL5aSeMqBr/miq1l9pmX2eBk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=Z9CKkjG3vxVnDuVHBP4vYmsiDKebLgcksrfYYlPK73o=; b=MZQ5Cazqn+eoopQcpr6AdWcZgIzXTEZNKv5Xc/L/giQI326FxOkV5B311F6AxwCbXF 3zYi8N+9AJzqm3s5BVkEOKrSLF8v0zo9+ya97vN+vi4y5B0bNTVXZb5NTS5X+KnNYOAG 30MpXGJM2gyov+vcES8XGSUI413s+ofzaL+ZlBJUltkl4rH5/OpRKULK+2SX3j4bV8sJ kwQ2xnvUy8lgqbRNi3gR297yQGpTGGN+VU4clmoIr1eI/59nEu1ggJWDAcEMead5w+8b cSilA2SrULphzQm68ppwnvnciwe8SyuWsXwAZmstXAeMUb9e9/tif5UA9Ool714O8dTg KGyA== X-Gm-Message-State: AOAM532cK6WFLMCl6JPM7KKqq8P8wpIufj/04eFAa8Yd7ZfxQS3Uuim0 8wMfTz8tnm1eFakYygcxral/A+QqmofAH67K4clJgZ5jI4aV9eNBeTGN4CM0UdPYdE62GHF4oWq XTj5vDpejr8qJOhKeBJ8HHHR00Q8ONqO8de56G8xxnNSTjFBiwupZ3sGxIiYWcxYTvURx4yR6vw ZlW1sUaQ== X-Google-Smtp-Source: ABdhPJwVhU3z0guHULOl6bjLP8k7wpeZbx6h7PNPC+7RbkfY/VUhJImUJsYMdJZUdmNL04c2LtWchw== X-Received: by 2002:a17:90b:3882:: with SMTP id mu2mr6364198pjb.170.1623781921376; Tue, 15 Jun 2021 11:32:01 -0700 (PDT) Received: from dev01.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id i125sm3803823pfc.7.2021.06.15.11.31.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 11:32:00 -0700 (PDT) From: Devesh Sharma To: linux-rdma@vger.kernel.org Cc: Devesh Sharma Subject: [PATCH V5 rdma-core 1/5] Update kernel headers Date: Wed, 16 Jun 2021 00:01:44 +0530 Message-Id: <20210615183148.1179042-2-devesh.sharma@broadcom.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210615183148.1179042-1-devesh.sharma@broadcom.com> References: <20210615183148.1179042-1-devesh.sharma@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org To commit ?? ("RDMA/bnxt_re: update ABI to pass wqe-mode to user space"). Signed-off-by: Devesh Sharma --- kernel-headers/rdma/bnxt_re-abi.h | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/kernel-headers/rdma/bnxt_re-abi.h b/kernel-headers/rdma/bnxt_re-abi.h index dc52e3cf..52205ed2 100644 --- a/kernel-headers/rdma/bnxt_re-abi.h +++ b/kernel-headers/rdma/bnxt_re-abi.h @@ -49,7 +49,8 @@ #define BNXT_RE_CHIP_ID0_CHIP_MET_SFT 0x18 enum { - BNXT_RE_UCNTX_CMASK_HAVE_CCTX = 0x1ULL + BNXT_RE_UCNTX_CMASK_HAVE_CCTX = 0x1ULL, + BNXT_RE_UCNTX_CMASK_HAVE_MODE = 0x02ULL }; struct bnxt_re_uctx_resp { @@ -62,6 +63,8 @@ struct bnxt_re_uctx_resp { __aligned_u64 comp_mask; __u32 chip_id0; __u32 chip_id1; + __u32 mode; + __u32 rsvd1; /* padding */ }; /* From patchwork Tue Jun 15 18:31:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 12322647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14699C48BE5 for ; Tue, 15 Jun 2021 18:32:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EECDD61185 for ; Tue, 15 Jun 2021 18:32:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231451AbhFOSeL (ORCPT ); Tue, 15 Jun 2021 14:34:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231202AbhFOSeI (ORCPT ); Tue, 15 Jun 2021 14:34:08 -0400 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55D52C061574 for ; Tue, 15 Jun 2021 11:32:04 -0700 (PDT) Received: by mail-pj1-x1031.google.com with SMTP id o88-20020a17090a0a61b029016eeb2adf66so2199125pjo.4 for ; Tue, 15 Jun 2021 11:32:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=NLmytu9MF2xJohkzCnT3Y8b1D4+3mYfVCoeVaAR9syk=; b=X7tm4hgh7nthM0c1of2ZJmEsXJZVzBeHRJWAlEdp+nO889da0lx9E2JOZOwGdPZAP6 zQZrOHBal6Bk7mNVMRchyp5OB/4uNtI3Cdev06E6B5QveVTHYX/rtcXalH2lB30F05z8 6GxNvycB8krlbsJft+O8JID7AQdTeU909kVEI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=NLmytu9MF2xJohkzCnT3Y8b1D4+3mYfVCoeVaAR9syk=; b=FSdhACafon9NJvC/g7Pg2Pw0II32wxvDct/puzUEFJUUQdp/AtueP+AFYMy1AwC8ch vp1RJnCItT+ZOmqOnfC+uvWaXz/qqePeq4NE2qgUYytWp6tx8vVfoZ870FhWD+b7QDTl pHm5O35Jf0jZOA3m1vNw3BuBTLsWFPsjKT8Cxc7Yaa4jh52kxO4BIIimoVM2pI4wd06W NzHpSxuFNV9DhE2knNGKaioBHG3C4Yp0TjdNh3OijZYO18dW+n2mYig6UdYIl7Erm+KY XRH1IHmH97eFo6zZOPjZ4EnyImoZfB1tgmxfwl0iz5plKuRRgvB6hh8IHqThl3bC+JmA bt8A== X-Gm-Message-State: AOAM533qoFM0UMK5c+guhnChPZVLW0U847mUJOS6UoN1lhXSVRqGWiYa DgX25/MG5BYzL9AyNbXMm5vcZOcnsBrTVVNnhzn4/8/e+TLYH4o9z6JEvhYFJAtLQ2ThHp/lxgU sY6obV8+YuqOXaCWcj6mrSNUcu9BvA3n0i7zmphesZs4TuTDQCALVe8kAbAIb88KOUpCyw4vPzs Zz9dxh1A== X-Google-Smtp-Source: ABdhPJwYYuIwJkriLT4Hj37iE8kKAOTk8vPufYPwbbdU64ti/7JVoAu4DEqEO3Dm9mDlDvd6hnFhBA== X-Received: by 2002:a17:902:ced1:b029:115:3e22:1eb4 with SMTP id d17-20020a170902ced1b02901153e221eb4mr5147934plg.19.1623781923108; Tue, 15 Jun 2021 11:32:03 -0700 (PDT) Received: from dev01.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id i125sm3803823pfc.7.2021.06.15.11.32.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 11:32:02 -0700 (PDT) From: Devesh Sharma To: linux-rdma@vger.kernel.org Cc: Devesh Sharma Subject: [PATCH V5 rdma-core 2/5] bnxt_re/lib: Read wqe mode from the driver Date: Wed, 16 Jun 2021 00:01:45 +0530 Message-Id: <20210615183148.1179042-3-devesh.sharma@broadcom.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210615183148.1179042-1-devesh.sharma@broadcom.com> References: <20210615183148.1179042-1-devesh.sharma@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org During bnxt_re device context creation, read the wqe mode and store it in context structure and in the QP context structure. wqe mode would be required to change between fixed size wqe and non-fixed sized wqe modes of SQ/RQ in gen-p5 or newer devices. Signed-off-by: Devesh Sharma --- providers/bnxt_re/bnxt_re-abi.h | 5 +++++ providers/bnxt_re/main.c | 4 ++++ providers/bnxt_re/main.h | 2 ++ providers/bnxt_re/verbs.c | 1 + 4 files changed, 12 insertions(+) diff --git a/providers/bnxt_re/bnxt_re-abi.h b/providers/bnxt_re/bnxt_re-abi.h index c82019e8..d138cd9c 100644 --- a/providers/bnxt_re/bnxt_re-abi.h +++ b/providers/bnxt_re/bnxt_re-abi.h @@ -196,6 +196,11 @@ enum bnxt_re_ud_cqe_mask { BNXT_RE_UD_CQE_SRCQPLO_SHIFT = 0x30 }; +enum bnxt_re_modes { + BNXT_RE_WQE_MODE_STATIC = 0x00, + BNXT_RE_WQE_MODE_VARIABLE = 0x01 +}; + struct bnxt_re_db_hdr { __le32 indx; __le32 typ_qid; /* typ: 4, qid:20*/ diff --git a/providers/bnxt_re/main.c b/providers/bnxt_re/main.c index 1779e1ec..ee9edd7d 100644 --- a/providers/bnxt_re/main.c +++ b/providers/bnxt_re/main.c @@ -158,6 +158,10 @@ static struct verbs_context *bnxt_re_alloc_context(struct ibv_device *vdev, BNXT_RE_CHIP_ID0_CHIP_MET_SFT) & 0xFF; } + + if (resp.comp_mask & BNXT_RE_UCNTX_CMASK_HAVE_MODE) + cntx->wqe_mode = resp.mode; + pthread_spin_init(&cntx->fqlock, PTHREAD_PROCESS_PRIVATE); /* mmap shared page. */ cntx->shpg = mmap(NULL, rdev->pg_size, PROT_READ | PROT_WRITE, diff --git a/providers/bnxt_re/main.h b/providers/bnxt_re/main.h index a63719e8..dc8166f2 100644 --- a/providers/bnxt_re/main.h +++ b/providers/bnxt_re/main.h @@ -146,6 +146,7 @@ struct bnxt_re_qp { uint64_t wqe_cnt; uint16_t mtu; uint16_t qpst; + uint32_t qpmode; uint8_t qptyp; /* irdord? */ }; @@ -178,6 +179,7 @@ struct bnxt_re_context { uint32_t max_srq; struct bnxt_re_dpi udpi; void *shpg; + uint32_t wqe_mode; pthread_mutex_t shlock; pthread_spinlock_t fqlock; }; diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index fb2cf5ac..11c01574 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -952,6 +952,7 @@ struct ibv_qp *bnxt_re_create_qp(struct ibv_pd *ibvpd, goto fail; /* alloc queues */ qp->cctx = &cntx->cctx; + qp->qpmode = cntx->wqe_mode & BNXT_RE_WQE_MODE_VARIABLE; if (bnxt_re_alloc_queues(qp, attr, dev->pg_size)) goto failq; /* Fill ibv_cmd */ From patchwork Tue Jun 15 18:31:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 12322649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CABE8C48BE8 for ; Tue, 15 Jun 2021 18:32:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ADDF261185 for ; Tue, 15 Jun 2021 18:32:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231260AbhFOSeM (ORCPT ); Tue, 15 Jun 2021 14:34:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231667AbhFOSeL (ORCPT ); Tue, 15 Jun 2021 14:34:11 -0400 Received: from mail-pl1-x636.google.com (mail-pl1-x636.google.com [IPv6:2607:f8b0:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44FF1C061574 for ; Tue, 15 Jun 2021 11:32:06 -0700 (PDT) Received: by mail-pl1-x636.google.com with SMTP id e1so8909084pld.13 for ; Tue, 15 Jun 2021 11:32:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=7bTY5xpEWMmzT80HcFQslEC986NYpK1Kqp4NGiKWwME=; b=EEGbZvpKMIUMjIO1O4G3O+eTYlvr8ILwawB7pErzHqIH/CV5ZAEBC4Z1gjrgdEJzmm ngNFhdzHsY+fvhSW+OO5A8Ugvl7uSYsMYEmq4JiZDOQplyQZO5wtCaG7YgijgS3sZvFw 3vOHhTbEEHUq/h4Tb7yUHEabMgXi56zsiK6BE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=7bTY5xpEWMmzT80HcFQslEC986NYpK1Kqp4NGiKWwME=; b=I61lb0nivHmVobhpo7C/jwGHcLveiUe2EUXcCqfr3dMDle5QYwJDYQ3T28M9zRFd9P TT8MkPOmciUJcv7KcpAvROmxxR6/PcShYc8/+THc5LGtIKYIMdbWZ+HlX0r3fcId6vK5 vnjnobiLwaHg1vyP2z2sQ+SC8OrWCROeM1lxJ04MNedPB9NUDyNb67Pgr43guRL2ppFe ANAD+KArkYUabt1la/t7FsaPbb0DY/wULR+rcyVSzAT4br5GhIovfU88XiE+kg5gZqgA pByPAnoHkVXdvmsF29QzwkKtAs58SfMzlL9Wwn6M+HmMGjP/hJOcaqtPtdlk4xzIpA7d CusQ== X-Gm-Message-State: AOAM533LAghL9RrI9ZxltpHTBSqtgAJVAHHev4oKBTutqlb4z7InwESo bJQ9pvoVCovSu449d1x9sBD4O20RkcFfTu6alqLGsEtUZ8W8lJAqhsRJuifUwxBlhPlvRZHBG8R kE2CCINclzZHO7tSsWhA9kSzCGDBqwtdwWuLLyoMrOwHFPxDmsLAy9iBKXR83qd+54wwD0PiqSl NuCUlChA== X-Google-Smtp-Source: ABdhPJzQluRJpJX19B1Ntw1dSXSDDuipDyRQvZMg5NkHk0akn4jFsW8dRrZj3THES6TywjfxXDGh2w== X-Received: by 2002:a17:902:f283:b029:10d:e7f3:725 with SMTP id k3-20020a170902f283b029010de7f30725mr3802501plc.45.1623781924830; Tue, 15 Jun 2021 11:32:04 -0700 (PDT) Received: from dev01.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id i125sm3803823pfc.7.2021.06.15.11.32.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 11:32:04 -0700 (PDT) From: Devesh Sharma To: linux-rdma@vger.kernel.org Cc: Devesh Sharma Subject: [PATCH V5 rdma-core 3/5] bnxt_re/lib: add a function to initialize software queue Date: Wed, 16 Jun 2021 00:01:46 +0530 Message-Id: <20210615183148.1179042-4-devesh.sharma@broadcom.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210615183148.1179042-1-devesh.sharma@broadcom.com> References: <20210615183148.1179042-1-devesh.sharma@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Splitting the shadow software queue initialization into a separate function. Same is being called for both RQ and SQ during create QP. Signed-off-by: Devesh Sharma --- providers/bnxt_re/main.h | 2 ++ providers/bnxt_re/verbs.c | 65 ++++++++++++++++++++++++--------------- 2 files changed, 43 insertions(+), 24 deletions(-) diff --git a/providers/bnxt_re/main.h b/providers/bnxt_re/main.h index dc8166f2..71a449e3 100644 --- a/providers/bnxt_re/main.h +++ b/providers/bnxt_re/main.h @@ -96,6 +96,8 @@ struct bnxt_re_wrid { uint64_t wrid; uint32_t bytes; int next_idx; + uint32_t st_slot_idx; + uint8_t slots; uint8_t sig; }; diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index 11c01574..b07628bb 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -847,9 +847,27 @@ static void bnxt_re_free_queues(struct bnxt_re_qp *qp) bnxt_re_free_aligned(qp->jsqq->hwque); } +static int bnxt_re_alloc_init_swque(struct bnxt_re_joint_queue *jqq, int nwr) +{ + int indx; + + jqq->swque = calloc(nwr, sizeof(struct bnxt_re_wrid)); + if (!jqq->swque) + return -ENOMEM; + jqq->start_idx = 0; + jqq->last_idx = nwr - 1; + for (indx = 0; indx < nwr; indx++) + jqq->swque[indx].next_idx = indx + 1; + jqq->swque[jqq->last_idx].next_idx = 0; + jqq->last_idx = 0; + + return 0; +} + static int bnxt_re_alloc_queues(struct bnxt_re_qp *qp, struct ibv_qp_init_attr *attr, - uint32_t pg_size) { + uint32_t pg_size) +{ struct bnxt_re_psns_ext *psns_ext; struct bnxt_re_wrid *swque; struct bnxt_re_queue *que; @@ -857,22 +875,23 @@ static int bnxt_re_alloc_queues(struct bnxt_re_qp *qp, uint32_t psn_depth; uint32_t psn_size; int ret, indx; + uint32_t nswr; que = qp->jsqq->hwque; que->stride = bnxt_re_get_sqe_sz(); /* 8916 adjustment */ - que->depth = roundup_pow_of_two(attr->cap.max_send_wr + 1 + - BNXT_RE_FULL_FLAG_DELTA); - que->diff = que->depth - attr->cap.max_send_wr; + nswr = roundup_pow_of_two(attr->cap.max_send_wr + 1 + + BNXT_RE_FULL_FLAG_DELTA); + que->diff = nswr - attr->cap.max_send_wr; /* psn_depth extra entries of size que->stride */ psn_size = bnxt_re_is_chip_gen_p5(qp->cctx) ? sizeof(struct bnxt_re_psns_ext) : sizeof(struct bnxt_re_psns); - psn_depth = (que->depth * psn_size) / que->stride; - if ((que->depth * psn_size) % que->stride) + psn_depth = (nswr * psn_size) / que->stride; + if ((nswr * psn_size) % que->stride) psn_depth++; - que->depth += psn_depth; + que->depth = nswr + psn_depth; /* PSN-search memory is allocated without checking for * QP-Type. Kenrel driver do not map this memory if it * is UD-qp. UD-qp use this memory to maintain WC-opcode. @@ -884,44 +903,42 @@ static int bnxt_re_alloc_queues(struct bnxt_re_qp *qp, /* exclude psns depth*/ que->depth -= psn_depth; /* start of spsn space sizeof(struct bnxt_re_psns) each. */ - psns = (que->va + que->stride * que->depth); + psns = (que->va + que->stride * nswr); psns_ext = (struct bnxt_re_psns_ext *)psns; - swque = calloc(que->depth, sizeof(struct bnxt_re_wrid)); - if (!swque) { + + ret = bnxt_re_alloc_init_swque(qp->jsqq, nswr); + if (ret) { ret = -ENOMEM; goto fail; } - for (indx = 0 ; indx < que->depth; indx++, psns++) + swque = qp->jsqq->swque; + for (indx = 0; indx < nswr; indx++, psns++) swque[indx].psns = psns; if (bnxt_re_is_chip_gen_p5(qp->cctx)) { - for (indx = 0 ; indx < que->depth; indx++, psns_ext++) { + for (indx = 0; indx < nswr; indx++, psns_ext++) { swque[indx].psns_ext = psns_ext; swque[indx].psns = (struct bnxt_re_psns *)psns_ext; } } - qp->jsqq->swque = swque; - - qp->cap.max_swr = que->depth; + qp->cap.max_swr = nswr; pthread_spin_init(&que->qlock, PTHREAD_PROCESS_PRIVATE); if (qp->jrqq) { que = qp->jrqq->hwque; que->stride = bnxt_re_get_rqe_sz(); - que->depth = roundup_pow_of_two(attr->cap.max_recv_wr + 1); - que->diff = que->depth - attr->cap.max_recv_wr; + nswr = roundup_pow_of_two(attr->cap.max_recv_wr + 1); + que->depth = nswr; + que->diff = nswr - attr->cap.max_recv_wr; ret = bnxt_re_alloc_aligned(que, pg_size); if (ret) goto fail; - pthread_spin_init(&que->qlock, PTHREAD_PROCESS_PRIVATE); /* For RQ only bnxt_re_wri.wrid is used. */ - qp->jrqq->swque = calloc(que->depth, - sizeof(struct bnxt_re_wrid)); - if (!qp->jrqq->swque) { - ret = -ENOMEM; + ret = bnxt_re_alloc_init_swque(qp->jrqq, nswr); + if (ret) goto fail; - } - qp->cap.max_rwr = que->depth; + pthread_spin_init(&que->qlock, PTHREAD_PROCESS_PRIVATE); + qp->cap.max_rwr = nswr; } return 0; From patchwork Tue Jun 15 18:31:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 12322651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76E88C49EA2 for ; Tue, 15 Jun 2021 18:32:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4E6D6610F7 for ; Tue, 15 Jun 2021 18:32:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231303AbhFOSeM (ORCPT ); Tue, 15 Jun 2021 14:34:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231208AbhFOSeM (ORCPT ); Tue, 15 Jun 2021 14:34:12 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8E01AC06175F for ; Tue, 15 Jun 2021 11:32:07 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id ei4so177257pjb.3 for ; Tue, 15 Jun 2021 11:32:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=U3ICardh+GcXllqe2JFBHsuducRA3snc2mfUcQlbISI=; b=QDAaFFBD/Y+AjzZVGqfvUfCuUC+aj9X2hYuaVqMpczNxKURaStsueS9HdpoE0ohETI R0qkdmD8o4usyeYj0fhgJYCL/wQMcLF41e+Pq2c/YXYqzdrI5/Ttjg6/i4OzKCcdbyo+ 2h18olSwgy4zlLc8s3hcV0qO6lR2iOa3xDVvI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=U3ICardh+GcXllqe2JFBHsuducRA3snc2mfUcQlbISI=; b=lY4ARVA2dHZcSp+ZwVjfJetLrJamcve3ZGml51pi6IKjoUK1R9t6EzSequwGZczbF9 pHfqZUvxlJYPbAzR7dy5xeF8dbt0mYg+nSLgqiQ9fn+t5gUOmwyeTEYoFcfE0Rg275yv hq9S7TTPN89xARcVeWv7bGbH/dmJQRpbFvN40Ba829Bcg4+5p9CQ3yDwLn83I+l0RDZL lypLbYqHJQk8+A4S8VXAUfx9YKqEUK4LSBzBy41SKm36BQYrTx13og8uv4PA+eigFZrG o3ySpeTZMT8V3ubplLeFYxGPe3pe/1lZYRozkqd20Vcfst2sSEEKdKkLv/7g00lNTSeL QCrQ== X-Gm-Message-State: AOAM532WOqPgRZ6GgHqQLI6mkX8yj29/ZEhgHYAa4rUw4rYb/fTLaUHZ o76i2airXWYAWY86opqO46H5VnDXKRMjIc1dhr14cFRPQmh+6qsH01F5WKEIAEoF46cIBC8r9o5 hy1mG0jfqNvJlCP3EgNczsK6JlD6L3QgdKuYF+qzN/93GAlA5YCgx7mD/ipq0Ss3p822SCWQynA 7ziu8LsA== X-Google-Smtp-Source: ABdhPJzRkokF99wFNkeKm7clTLt6rDwlVGBQJ11u3eSXEAoZnC6rhqAcgM2TD9KAekE/dXkvvPkWYg== X-Received: by 2002:a17:902:c613:b029:107:ce4:f7b9 with SMTP id r19-20020a170902c613b02901070ce4f7b9mr5163782plr.11.1623781926551; Tue, 15 Jun 2021 11:32:06 -0700 (PDT) Received: from dev01.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id i125sm3803823pfc.7.2021.06.15.11.32.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 11:32:06 -0700 (PDT) From: Devesh Sharma To: linux-rdma@vger.kernel.org Cc: Devesh Sharma Subject: [PATCH V5 rdma-core 4/5] bnxt_re/lib: Use separate indices for shadow queue Date: Wed, 16 Jun 2021 00:01:47 +0530 Message-Id: <20210615183148.1179042-5-devesh.sharma@broadcom.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210615183148.1179042-1-devesh.sharma@broadcom.com> References: <20210615183148.1179042-1-devesh.sharma@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The shadow queue is used for wrid and flush wqe management. The indices used in this queue are independent to what is actually used by hardware. Thus, detaching the shadow queue indices from hardware queue indices. This even more useful when the hardware queue indices has alignment other than wqe boundary. Signed-off-by: Devesh Sharma --- providers/bnxt_re/main.h | 20 ++++++ providers/bnxt_re/memory.h | 8 +-- providers/bnxt_re/verbs.c | 128 ++++++++++++++++++++++--------------- 3 files changed, 101 insertions(+), 55 deletions(-) diff --git a/providers/bnxt_re/main.h b/providers/bnxt_re/main.h index 71a449e3..5d05dd85 100644 --- a/providers/bnxt_re/main.h +++ b/providers/bnxt_re/main.h @@ -436,4 +436,24 @@ static inline void bnxt_re_change_cq_phase(struct bnxt_re_cq *cq) if (!cq->cqq.head) cq->phase = (~cq->phase & BNXT_RE_BCQE_PH_MASK); } + +static inline void *bnxt_re_get_swqe(struct bnxt_re_joint_queue *jqq, + uint32_t *wqe_idx) +{ + if (wqe_idx) + *wqe_idx = jqq->start_idx; + return &jqq->swque[jqq->start_idx]; +} + +static inline void bnxt_re_jqq_mod_start(struct bnxt_re_joint_queue *jqq, + uint32_t idx) +{ + jqq->start_idx = jqq->swque[idx].next_idx; +} + +static inline void bnxt_re_jqq_mod_last(struct bnxt_re_joint_queue *jqq, + uint32_t idx) +{ + jqq->last_idx = jqq->swque[idx].next_idx; +} #endif diff --git a/providers/bnxt_re/memory.h b/providers/bnxt_re/memory.h index 75564c43..5bcdef9a 100644 --- a/providers/bnxt_re/memory.h +++ b/providers/bnxt_re/memory.h @@ -97,14 +97,14 @@ static inline uint32_t bnxt_re_incr(uint32_t val, uint32_t max) return (++val & (max - 1)); } -static inline void bnxt_re_incr_tail(struct bnxt_re_queue *que) +static inline void bnxt_re_incr_tail(struct bnxt_re_queue *que, uint8_t cnt) { - que->tail = bnxt_re_incr(que->tail, que->depth); + que->tail = (que->tail + cnt) & (que->depth - 1); } -static inline void bnxt_re_incr_head(struct bnxt_re_queue *que) +static inline void bnxt_re_incr_head(struct bnxt_re_queue *que, uint8_t cnt) { - que->head = bnxt_re_incr(que->head, que->depth); + que->head = (que->head + cnt) & (que->depth - 1); } #endif diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index b07628bb..fb097308 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -247,10 +247,12 @@ static uint8_t bnxt_re_poll_err_scqe(struct bnxt_re_qp *qp, struct bnxt_re_wrid *swrid; struct bnxt_re_psns *spsn; struct bnxt_re_cq *scq; - uint32_t head = sq->head; uint8_t status; + uint32_t head; scq = to_bnxt_re_cq(qp->ibvqp.send_cq); + + head = qp->jsqq->last_idx; cntx = to_bnxt_re_context(scq->ibvcq.context); swrid = &qp->jsqq->swque[head]; spsn = swrid->psns; @@ -267,7 +269,8 @@ static uint8_t bnxt_re_poll_err_scqe(struct bnxt_re_qp *qp, BNXT_RE_PSNS_OPCD_MASK; ibvwc->byte_len = 0; - bnxt_re_incr_head(sq); + bnxt_re_incr_head(sq, swrid->slots); + bnxt_re_jqq_mod_last(qp->jsqq, head); if (qp->qpst != IBV_QPS_ERR) qp->qpst = IBV_QPS_ERR; @@ -287,13 +290,14 @@ static uint8_t bnxt_re_poll_success_scqe(struct bnxt_re_qp *qp, struct bnxt_re_queue *sq = qp->jsqq->hwque; struct bnxt_re_wrid *swrid; struct bnxt_re_psns *spsn; - uint32_t head = sq->head; uint8_t pcqe = false; uint32_t cindx; + uint32_t head; + head = qp->jsqq->last_idx; swrid = &qp->jsqq->swque[head]; spsn = swrid->psns; - cindx = le32toh(scqe->con_indx); + cindx = le32toh(scqe->con_indx) & (qp->cap.max_swr - 1); if (!(swrid->sig & IBV_SEND_SIGNALED)) { *cnt = 0; @@ -313,8 +317,10 @@ static uint8_t bnxt_re_poll_success_scqe(struct bnxt_re_qp *qp, *cnt = 1; } - bnxt_re_incr_head(sq); - if (sq->head != cindx) + bnxt_re_incr_head(sq, swrid->slots); + bnxt_re_jqq_mod_last(qp->jsqq, head); + + if (qp->jsqq->last_idx != cindx) pcqe = true; return pcqe; @@ -352,23 +358,29 @@ static void bnxt_re_release_srqe(struct bnxt_re_srq *srq, int tag) static int bnxt_re_poll_err_rcqe(struct bnxt_re_qp *qp, struct ibv_wc *ibvwc, struct bnxt_re_bcqe *hdr, void *cqe) { + struct bnxt_re_context *cntx; + struct bnxt_re_wrid *swque; struct bnxt_re_queue *rq; + uint8_t status, cnt = 0; struct bnxt_re_cq *rcq; - struct bnxt_re_context *cntx; - uint8_t status; + uint32_t head = 0; rcq = to_bnxt_re_cq(qp->ibvqp.recv_cq); cntx = to_bnxt_re_context(rcq->ibvcq.context); if (!qp->srq) { rq = qp->jrqq->hwque; - ibvwc->wr_id = qp->jrqq->swque[rq->head].wrid; + head = qp->jrqq->last_idx; + swque = &qp->jrqq->swque[head]; + ibvwc->wr_id = swque->wrid; + cnt = swque->slots; } else { struct bnxt_re_srq *srq; int tag; srq = qp->srq; rq = srq->srqq; + cnt = 1; tag = le32toh(hdr->qphi_rwrid) & BNXT_RE_BCQE_RWRID_MASK; ibvwc->wr_id = srq->srwrid[tag].wrid; bnxt_re_release_srqe(srq, tag); @@ -387,7 +399,10 @@ static int bnxt_re_poll_err_rcqe(struct bnxt_re_qp *qp, struct ibv_wc *ibvwc, ibvwc->wc_flags = 0; if (qp->qptyp == IBV_QPT_UD) ibvwc->src_qp = 0; - bnxt_re_incr_head(rq); + + if (!qp->srq) + bnxt_re_jqq_mod_last(qp->jrqq, head); + bnxt_re_incr_head(rq, cnt); if (!qp->srq) { pthread_spin_lock(&cntx->fqlock); @@ -417,14 +432,20 @@ static void bnxt_re_poll_success_rcqe(struct bnxt_re_qp *qp, struct ibv_wc *ibvwc, struct bnxt_re_bcqe *hdr, void *cqe) { - struct bnxt_re_queue *rq; - struct bnxt_re_rc_cqe *rcqe; uint8_t flags, is_imm, is_rdma; + struct bnxt_re_rc_cqe *rcqe; + struct bnxt_re_wrid *swque; + struct bnxt_re_queue *rq; + uint32_t head = 0; + uint8_t cnt = 0; rcqe = cqe; if (!qp->srq) { rq = qp->jrqq->hwque; - ibvwc->wr_id = qp->jrqq->swque[rq->head].wrid; + head = qp->jrqq->last_idx; + swque = &qp->jrqq->swque[head]; + ibvwc->wr_id = swque->wrid; + cnt = swque->slots; } else { struct bnxt_re_srq *srq; int tag; @@ -433,6 +454,7 @@ static void bnxt_re_poll_success_rcqe(struct bnxt_re_qp *qp, rq = srq->srqq; tag = le32toh(hdr->qphi_rwrid) & BNXT_RE_BCQE_RWRID_MASK; ibvwc->wr_id = srq->srwrid[tag].wrid; + cnt = 1; bnxt_re_release_srqe(srq, tag); } @@ -463,7 +485,9 @@ static void bnxt_re_poll_success_rcqe(struct bnxt_re_qp *qp, if (qp->qptyp == IBV_QPT_UD) bnxt_re_fill_ud_cqe(ibvwc, hdr, cqe); - bnxt_re_incr_head(rq); + if (!qp->srq) + bnxt_re_jqq_mod_last(qp->jrqq, head); + bnxt_re_incr_head(rq, cnt); } static uint8_t bnxt_re_poll_rcqe(struct bnxt_re_qp *qp, struct ibv_wc *ibvwc, @@ -575,7 +599,7 @@ static int bnxt_re_poll_one(struct bnxt_re_cq *cq, int nwc, struct ibv_wc *wc) *qp_handle = 0x0ULL; /* mark cqe as read */ qp_handle = NULL; } - bnxt_re_incr_head(&cq->cqq); + bnxt_re_incr_head(&cq->cqq, 1); bnxt_re_change_cq_phase(cq); skipp_real: if (cnt) { @@ -592,21 +616,21 @@ skipp_real: return dqed; } -static int bnxt_re_poll_flush_wcs(struct bnxt_re_queue *que, - struct bnxt_re_wrid *wridp, +static int bnxt_re_poll_flush_wcs(struct bnxt_re_joint_queue *jqq, struct ibv_wc *ibvwc, uint32_t qpid, int nwc) { + uint8_t opcode = IBV_WC_RECV; + struct bnxt_re_queue *que; struct bnxt_re_wrid *wrid; struct bnxt_re_psns *psns; - uint32_t cnt = 0, head; - uint8_t opcode = IBV_WC_RECV; + uint32_t cnt = 0; + que = jqq->hwque; while (nwc) { if (bnxt_re_is_que_empty(que)) break; - head = que->head; - wrid = &wridp[head]; + wrid = &jqq->swque[jqq->last_idx]; if (wrid->psns) { psns = wrid->psns; opcode = (le32toh(psns->opc_spsn) >> @@ -621,7 +645,8 @@ static int bnxt_re_poll_flush_wcs(struct bnxt_re_queue *que, ibvwc->byte_len = 0; ibvwc->wc_flags = 0; - bnxt_re_incr_head(que); + bnxt_re_jqq_mod_last(jqq, jqq->last_idx); + bnxt_re_incr_head(que, wrid->slots); nwc--; cnt++; ibvwc++; @@ -636,8 +661,7 @@ static int bnxt_re_poll_flush_wqes(struct bnxt_re_cq *cq, int32_t nwc) { struct bnxt_re_fque_node *cur, *tmp; - struct bnxt_re_wrid *wridp; - struct bnxt_re_queue *que; + struct bnxt_re_joint_queue *jqq; struct bnxt_re_qp *qp; bool sq_list = false; uint32_t polled = 0; @@ -648,18 +672,15 @@ static int bnxt_re_poll_flush_wqes(struct bnxt_re_cq *cq, if (sq_list) { qp = container_of(cur, struct bnxt_re_qp, snode); - que = qp->jsqq->hwque; - wridp = qp->jsqq->swque; + jqq = qp->jsqq; } else { qp = container_of(cur, struct bnxt_re_qp, rnode); - que = qp->jrqq->hwque; - wridp = qp->jrqq->swque; + jqq = qp->jrqq; } - if (bnxt_re_is_que_empty(que)) + if (bnxt_re_is_que_empty(jqq->hwque)) continue; - polled += bnxt_re_poll_flush_wcs(que, wridp, - ibvwc + polled, + polled += bnxt_re_poll_flush_wcs(jqq, ibvwc + polled, qp->qpid, nwc - polled); if (!(nwc - polled)) @@ -1165,14 +1186,17 @@ static void bnxt_re_fill_psns(struct bnxt_re_qp *qp, struct bnxt_re_wrid *wrid, psns_ext->st_slot_idx = 0; } -static void bnxt_re_fill_wrid(struct bnxt_re_wrid *wrid, struct ibv_send_wr *wr, - uint32_t len, uint8_t sqsig) +static void bnxt_re_fill_wrid(struct bnxt_re_wrid *wrid, uint64_t wr_id, + uint32_t len, uint8_t sqsig, uint32_t st_idx, + uint8_t slots) { - wrid->wrid = wr->wr_id; + wrid->wrid = wr_id; wrid->bytes = len; wrid->sig = 0; - if (wr->send_flags & IBV_SEND_SIGNALED || sqsig) + if (sqsig) wrid->sig = IBV_SEND_SIGNALED; + wrid->st_slot_idx = st_idx; + wrid->slots = slots; } static int bnxt_re_build_send_sqe(struct bnxt_re_qp *qp, void *wqe, @@ -1291,6 +1315,8 @@ int bnxt_re_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, struct bnxt_re_bsqe *hdr; int ret = 0, bytes = 0; bool ring_db = false; + uint32_t swq_idx; + uint32_t sig; void *sqe; pthread_spin_lock(&sq->qlock); @@ -1317,8 +1343,6 @@ int bnxt_re_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, } sqe = (void *)(sq->va + (sq->tail * sq->stride)); - wrid = &qp->jsqq->swque[sq->tail]; - memset(sqe, 0, bnxt_re_get_sqe_sz()); hdr = sqe; is_inline = bnxt_re_set_hdr_flags(hdr, wr->send_flags, @@ -1366,9 +1390,12 @@ int bnxt_re_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, break; } - bnxt_re_fill_wrid(wrid, wr, bytes, qp->cap.sqsig); + wrid = bnxt_re_get_swqe(qp->jsqq, &swq_idx); + sig = ((wr->send_flags & IBV_SEND_SIGNALED) || qp->cap.sqsig); + bnxt_re_fill_wrid(wrid, wr->wr_id, bytes, sig, sq->tail, 1); bnxt_re_fill_psns(qp, wrid, wr->opcode, bytes); - bnxt_re_incr_tail(sq); + bnxt_re_jqq_mod_start(qp->jsqq, swq_idx); + bnxt_re_incr_tail(sq, 1); qp->wqe_cnt++; wr = wr->next; ring_db = true; @@ -1395,16 +1422,14 @@ bad_wr: } static int bnxt_re_build_rqe(struct bnxt_re_qp *qp, struct ibv_recv_wr *wr, - void *rqe) + void *rqe, uint32_t idx) { struct bnxt_re_brqe *hdr = rqe; - struct bnxt_re_wrid *wrid; struct bnxt_re_sge *sge; int wqe_sz, len; uint32_t hdrval; sge = (rqe + bnxt_re_get_rqe_hdr_sz()); - wrid = &qp->jrqq->swque[qp->jrqq->hwque->tail]; len = bnxt_re_build_sge(sge, wr->sg_list, wr->num_sge, false); wqe_sz = wr->num_sge + (bnxt_re_get_rqe_hdr_sz() >> 4); /* 16B align */ @@ -1416,12 +1441,7 @@ static int bnxt_re_build_rqe(struct bnxt_re_qp *qp, struct ibv_recv_wr *wr, hdrval = BNXT_RE_WR_OPCD_RECV; hdrval |= ((wqe_sz & BNXT_RE_HDR_WS_MASK) << BNXT_RE_HDR_WS_SHIFT); hdr->rsv_ws_fl_wt = htole32(hdrval); - hdr->wrid = htole32(qp->jrqq->hwque->tail); - - /* Fill wrid */ - wrid->wrid = wr->wr_id; - wrid->bytes = len; /* N.A. for RQE */ - wrid->sig = 0; /* N.A. for RQE */ + hdr->wrid = htole32(idx); return len; } @@ -1431,6 +1451,8 @@ int bnxt_re_post_recv(struct ibv_qp *ibvqp, struct ibv_recv_wr *wr, { struct bnxt_re_qp *qp = to_bnxt_re_qp(ibvqp); struct bnxt_re_queue *rq = qp->jrqq->hwque; + struct bnxt_re_wrid *swque; + uint32_t swq_idx; void *rqe; int ret; @@ -1452,14 +1474,18 @@ int bnxt_re_post_recv(struct ibv_qp *ibvqp, struct ibv_recv_wr *wr, rqe = (void *)(rq->va + (rq->tail * rq->stride)); memset(rqe, 0, bnxt_re_get_rqe_sz()); - ret = bnxt_re_build_rqe(qp, wr, rqe); + swque = bnxt_re_get_swqe(qp->jrqq, &swq_idx); + ret = bnxt_re_build_rqe(qp, wr, rqe, swq_idx); if (ret < 0) { pthread_spin_unlock(&rq->qlock); *bad = wr; return ENOMEM; } - bnxt_re_incr_tail(rq); + swque = bnxt_re_get_swqe(qp->jrqq, NULL); + bnxt_re_fill_wrid(swque, wr->wr_id, ret, 0, rq->tail, 1); + bnxt_re_jqq_mod_start(qp->jrqq, swq_idx); + bnxt_re_incr_tail(rq, 1); wr = wr->next; bnxt_re_ring_rq_db(qp); } @@ -1667,7 +1693,7 @@ int bnxt_re_post_srq_recv(struct ibv_srq *ibvsrq, struct ibv_recv_wr *wr, } srq->start_idx = srq->srwrid[srq->start_idx].next_idx; - bnxt_re_incr_tail(rq); + bnxt_re_incr_tail(rq, 1); wr = wr->next; bnxt_re_ring_srq_db(srq); count++; From patchwork Tue Jun 15 18:31:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Devesh Sharma X-Patchwork-Id: 12322653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EBAEC48BE5 for ; Tue, 15 Jun 2021 18:32:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 54FF0610F7 for ; Tue, 15 Jun 2021 18:32:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231310AbhFOSeO (ORCPT ); Tue, 15 Jun 2021 14:34:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231208AbhFOSeO (ORCPT ); Tue, 15 Jun 2021 14:34:14 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F859C061574 for ; Tue, 15 Jun 2021 11:32:09 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id z3-20020a17090a3983b029016bc232e40bso181174pjb.4 for ; Tue, 15 Jun 2021 11:32:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version; bh=jufeS3gErMO3/Nw8uDXjThHmzRd47fDdOqOONNq4vXY=; b=OsQsBvaT65cT8QY5uSb0P27h6+kUX1sUT013qNU0grzyHEGq1XvYuOjUB1n2hzLIC4 E6m46hkmnsXJNBOVq6+XT4Rv8ux5hxZQiuCOA2+B4X/RANXIF2A86ZZNACwTou8TC3nW +/Lfz/xV6cuM6G4IUfGI0ZqOMGH+UqgHXTt1A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version; bh=jufeS3gErMO3/Nw8uDXjThHmzRd47fDdOqOONNq4vXY=; b=q106tZhZpB70yw9yQqMB1/L9fci6UflfnQti15yx9BM9dIIbtLKg0i+QkXBZ/JQlOS SOfz9CeXB9PxsU9kqp98/MQrMECr14K+GDw1PoN11Z2K5fhunYUtevY8PADYEb6uIOsD Uq967ImdyqCRJI8smM7E1kYjnC8CvYbIymICprDjsOhPC0C1q4+XprxQJPau9isM6/71 iEa+Do9p7AEGHZ4HaaopggeYYWxYS5QhDoqUEvEJtu2f4qDAscsY/MPwUK1+efv02tLi yPWvx24w/QU6pm+vAmv9oECdiKWB9GjWTuQUILw62Mk4jDmKlaLx242ho4AIoA4jEXyS 6xRA== X-Gm-Message-State: AOAM531fo+33L4oz5xxS2ktqxh/YtkRRXL6jV5VMWB4rT8hCPajop3oz OWI49iEV4pK3YtLhtIrFqLEFBgAu9Gq3HCcLlRLz3jEE2xRfNXBdfqkAON0QYdF5whdWZR6vyZg oFcBZIh4JDBm6/OujkY+xq5wSWIgOFMU/jZlfjGo/neNWapqmWMDiFLdxxJ2NiD+hj+1e9E3hoj YQ7bQhjA== X-Google-Smtp-Source: ABdhPJyW6fI/I1cC6aQfvifNXN+fFFDaklVDtYqxdJ66DnI6vWqzBKwhY7g3iYWHNuw3M2uyDVtL1Q== X-Received: by 2002:a17:90b:3004:: with SMTP id hg4mr6444566pjb.12.1623781928298; Tue, 15 Jun 2021 11:32:08 -0700 (PDT) Received: from dev01.dhcp.broadcom.net ([192.19.234.250]) by smtp.gmail.com with ESMTPSA id i125sm3803823pfc.7.2021.06.15.11.32.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Jun 2021 11:32:07 -0700 (PDT) From: Devesh Sharma To: linux-rdma@vger.kernel.org Cc: Devesh Sharma Subject: [PATCH V5 rdma-core 5/5] bnxt_re/lib: Move hardware queue to 16B aligned indices Date: Wed, 16 Jun 2021 00:01:48 +0530 Message-Id: <20210615183148.1179042-6-devesh.sharma@broadcom.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210615183148.1179042-1-devesh.sharma@broadcom.com> References: <20210615183148.1179042-1-devesh.sharma@broadcom.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Move SQ and RQ indices from WQE boundary to 16B boundary alignment. Changing the SQ-wqe posting algorithm accordingly. The new alignment needs to pull a 16B slot from the hardware queue and initialize the current 16B into the hardware buffer. Depending on the max possible wqe size supported by hardware, the number of 16B slots are calculated and pulled for initialization. Currently 128B wqe is supported and it requires 8 slots. Signed-off-by: Devesh Sharma --- providers/bnxt_re/db.c | 10 +- providers/bnxt_re/main.h | 1 + providers/bnxt_re/memory.h | 33 +++- providers/bnxt_re/verbs.c | 369 ++++++++++++++++++++++++++----------- 4 files changed, 291 insertions(+), 122 deletions(-) diff --git a/providers/bnxt_re/db.c b/providers/bnxt_re/db.c index 3c797573..e99b7b62 100644 --- a/providers/bnxt_re/db.c +++ b/providers/bnxt_re/db.c @@ -62,18 +62,20 @@ static void bnxt_re_init_db_hdr(struct bnxt_re_db_hdr *hdr, uint32_t indx, void bnxt_re_ring_rq_db(struct bnxt_re_qp *qp) { struct bnxt_re_db_hdr hdr; + uint32_t tail; - bnxt_re_init_db_hdr(&hdr, qp->jrqq->hwque->tail, - qp->qpid, BNXT_RE_QUE_TYPE_RQ); + tail = qp->jrqq->hwque->tail / qp->jrqq->hwque->max_slots; + bnxt_re_init_db_hdr(&hdr, tail, qp->qpid, BNXT_RE_QUE_TYPE_RQ); bnxt_re_ring_db(qp->udpi, &hdr); } void bnxt_re_ring_sq_db(struct bnxt_re_qp *qp) { struct bnxt_re_db_hdr hdr; + uint32_t tail; - bnxt_re_init_db_hdr(&hdr, qp->jsqq->hwque->tail, - qp->qpid, BNXT_RE_QUE_TYPE_SQ); + tail = qp->jsqq->hwque->tail / qp->jsqq->hwque->max_slots; + bnxt_re_init_db_hdr(&hdr, tail, qp->qpid, BNXT_RE_QUE_TYPE_SQ); bnxt_re_ring_db(qp->udpi, &hdr); } diff --git a/providers/bnxt_re/main.h b/providers/bnxt_re/main.h index 5d05dd85..2e6131ae 100644 --- a/providers/bnxt_re/main.h +++ b/providers/bnxt_re/main.h @@ -44,6 +44,7 @@ #include #include #include +#include #include #include diff --git a/providers/bnxt_re/memory.h b/providers/bnxt_re/memory.h index 5bcdef9a..35fb057e 100644 --- a/providers/bnxt_re/memory.h +++ b/providers/bnxt_re/memory.h @@ -57,6 +57,8 @@ struct bnxt_re_queue { * and the consumer indices in the queue */ uint32_t diff; + uint32_t esize; + uint32_t max_slots; pthread_spinlock_t qlock; }; @@ -82,29 +84,44 @@ int bnxt_re_alloc_aligned(struct bnxt_re_queue *que, uint32_t pg_size); void bnxt_re_free_aligned(struct bnxt_re_queue *que); /* Basic queue operation */ -static inline uint32_t bnxt_re_is_que_full(struct bnxt_re_queue *que) +static inline void *bnxt_re_get_hwqe(struct bnxt_re_queue *que, uint32_t idx) { - return (((que->diff + que->tail) & (que->depth - 1)) == que->head); + idx += que->tail; + if (idx >= que->depth) + idx -= que->depth; + return (void *)(que->va + (idx << 4)); } -static inline uint32_t bnxt_re_is_que_empty(struct bnxt_re_queue *que) +static inline bool bnxt_re_is_que_full(struct bnxt_re_queue *que, + uint32_t slots) { - return que->tail == que->head; + int32_t avail, head, tail; + + head = que->head; + tail = que->tail; + avail = head - tail; + if (head <= tail) + avail += que->depth; + return avail <= (slots + que->diff); } -static inline uint32_t bnxt_re_incr(uint32_t val, uint32_t max) +static inline bool bnxt_re_is_que_empty(struct bnxt_re_queue *que) { - return (++val & (max - 1)); + return que->tail == que->head; } static inline void bnxt_re_incr_tail(struct bnxt_re_queue *que, uint8_t cnt) { - que->tail = (que->tail + cnt) & (que->depth - 1); + que->tail += cnt; + if (que->tail >= que->depth) + que->tail %= que->depth; } static inline void bnxt_re_incr_head(struct bnxt_re_queue *que, uint8_t cnt) { - que->head = (que->head + cnt) & (que->depth - 1); + que->head += cnt; + if (que->head >= que->depth) + que->head %= que->depth; } #endif diff --git a/providers/bnxt_re/verbs.c b/providers/bnxt_re/verbs.c index fb097308..d5dcbf51 100644 --- a/providers/bnxt_re/verbs.c +++ b/providers/bnxt_re/verbs.c @@ -885,7 +885,76 @@ static int bnxt_re_alloc_init_swque(struct bnxt_re_joint_queue *jqq, int nwr) return 0; } -static int bnxt_re_alloc_queues(struct bnxt_re_qp *qp, +static int bnxt_re_calc_wqe_sz(int nsge) +{ + /* This is used for both sq and rq. In case hdr size differs + * in future move to individual functions. + */ + return sizeof(struct bnxt_re_sge) * nsge + bnxt_re_get_sqe_hdr_sz(); +} + +static int bnxt_re_get_rq_slots(struct bnxt_re_dev *rdev, + struct bnxt_re_qp *qp, uint32_t nrwr, + uint32_t nsge) +{ + uint32_t max_wqesz; + uint32_t wqe_size; + uint32_t stride; + + stride = sizeof(struct bnxt_re_sge); + max_wqesz = bnxt_re_calc_wqe_sz(rdev->devattr.max_sge); + + wqe_size = bnxt_re_calc_wqe_sz(nsge); + if (wqe_size > max_wqesz) + return -EINVAL; + + if (qp->qpmode == BNXT_RE_WQE_MODE_STATIC) + wqe_size = bnxt_re_calc_wqe_sz(6); + + qp->jrqq->hwque->esize = wqe_size; + qp->jrqq->hwque->max_slots = wqe_size / stride; + + return (nrwr * wqe_size) / stride; +} + +static int bnxt_re_get_sq_slots(struct bnxt_re_dev *rdev, + struct bnxt_re_qp *qp, uint32_t nswr, + uint32_t nsge, uint32_t *ils) +{ + uint32_t max_wqesz; + uint32_t wqe_size; + uint32_t cal_ils; + uint32_t stride; + uint32_t ilsize; + uint32_t hdr_sz; + + hdr_sz = bnxt_re_get_sqe_hdr_sz(); + stride = sizeof(struct bnxt_re_sge); + max_wqesz = bnxt_re_calc_wqe_sz(rdev->devattr.max_sge); + ilsize = get_aligned(*ils, hdr_sz); + + wqe_size = bnxt_re_calc_wqe_sz(nsge); + if (ilsize) { + cal_ils = hdr_sz + ilsize; + wqe_size = MAX(cal_ils, wqe_size); + wqe_size = get_aligned(wqe_size, hdr_sz); + } + if (wqe_size > max_wqesz) + return -EINVAL; + + if (qp->qpmode == BNXT_RE_WQE_MODE_STATIC) + wqe_size = bnxt_re_calc_wqe_sz(6); + + if (*ils) + *ils = wqe_size - hdr_sz; + qp->jsqq->hwque->esize = wqe_size; + qp->jsqq->hwque->max_slots = (qp->qpmode == BNXT_RE_WQE_MODE_STATIC) ? + wqe_size / stride : 1; + return (nswr * wqe_size) / stride; +} + +static int bnxt_re_alloc_queues(struct bnxt_re_dev *dev, + struct bnxt_re_qp *qp, struct ibv_qp_init_attr *attr, uint32_t pg_size) { @@ -893,17 +962,27 @@ static int bnxt_re_alloc_queues(struct bnxt_re_qp *qp, struct bnxt_re_wrid *swque; struct bnxt_re_queue *que; struct bnxt_re_psns *psns; + uint32_t nswr, diff; uint32_t psn_depth; uint32_t psn_size; + uint32_t nsge; int ret, indx; - uint32_t nswr; + int nslots; que = qp->jsqq->hwque; - que->stride = bnxt_re_get_sqe_sz(); - /* 8916 adjustment */ - nswr = roundup_pow_of_two(attr->cap.max_send_wr + 1 + - BNXT_RE_FULL_FLAG_DELTA); - que->diff = nswr - attr->cap.max_send_wr; + diff = (qp->qpmode == BNXT_RE_WQE_MODE_VARIABLE) ? + 0 : BNXT_RE_FULL_FLAG_DELTA; + nswr = roundup_pow_of_two(attr->cap.max_send_wr + 1 + diff); + nsge = attr->cap.max_send_sge; + if (nsge % 2) + nsge++; + nslots = bnxt_re_get_sq_slots(dev, qp, nswr, nsge, + &attr->cap.max_inline_data); + if (nslots < 0) + return nslots; + que->stride = sizeof(struct bnxt_re_sge); + que->depth = nslots; + que->diff = (diff * que->esize) / que->stride; /* psn_depth extra entries of size que->stride */ psn_size = bnxt_re_is_chip_gen_p5(qp->cctx) ? @@ -912,7 +991,7 @@ static int bnxt_re_alloc_queues(struct bnxt_re_qp *qp, psn_depth = (nswr * psn_size) / que->stride; if ((nswr * psn_size) % que->stride) psn_depth++; - que->depth = nswr + psn_depth; + que->depth += psn_depth; /* PSN-search memory is allocated without checking for * QP-Type. Kenrel driver do not map this memory if it * is UD-qp. UD-qp use this memory to maintain WC-opcode. @@ -924,7 +1003,7 @@ static int bnxt_re_alloc_queues(struct bnxt_re_qp *qp, /* exclude psns depth*/ que->depth -= psn_depth; /* start of spsn space sizeof(struct bnxt_re_psns) each. */ - psns = (que->va + que->stride * nswr); + psns = (que->va + que->stride * que->depth); psns_ext = (struct bnxt_re_psns_ext *)psns; ret = bnxt_re_alloc_init_swque(qp->jsqq, nswr); @@ -947,10 +1026,19 @@ static int bnxt_re_alloc_queues(struct bnxt_re_qp *qp, if (qp->jrqq) { que = qp->jrqq->hwque; - que->stride = bnxt_re_get_rqe_sz(); nswr = roundup_pow_of_two(attr->cap.max_recv_wr + 1); - que->depth = nswr; - que->diff = nswr - attr->cap.max_recv_wr; + nsge = attr->cap.max_recv_sge; + if (nsge % 2) + nsge++; + nslots = bnxt_re_get_rq_slots(dev, qp, nswr, nsge); + if (nslots < 0) { + ret = nslots; + goto fail; + } + que->stride = sizeof(struct bnxt_re_sge); + que->depth = nslots; + que->diff = 0; + ret = bnxt_re_alloc_aligned(que, pg_size); if (ret) goto fail; @@ -971,10 +1059,10 @@ fail: struct ibv_qp *bnxt_re_create_qp(struct ibv_pd *ibvpd, struct ibv_qp_init_attr *attr) { - struct bnxt_re_qp *qp; - struct ubnxt_re_qp req; struct ubnxt_re_qp_resp resp; struct bnxt_re_qpcap *cap; + struct ubnxt_re_qp req; + struct bnxt_re_qp *qp; struct bnxt_re_context *cntx = to_bnxt_re_context(ibvpd->context); struct bnxt_re_dev *dev = to_bnxt_re_dev(cntx->ibvctx.context.device); @@ -991,7 +1079,7 @@ struct ibv_qp *bnxt_re_create_qp(struct ibv_pd *ibvpd, /* alloc queues */ qp->cctx = &cntx->cctx; qp->qpmode = cntx->wqe_mode & BNXT_RE_WQE_MODE_VARIABLE; - if (bnxt_re_alloc_queues(qp, attr, dev->pg_size)) + if (bnxt_re_alloc_queues(dev, qp, attr, dev->pg_size)) goto failq; /* Fill ibv_cmd */ cap = &qp->cap; @@ -1095,8 +1183,44 @@ int bnxt_re_destroy_qp(struct ibv_qp *ibvqp) return 0; } -static inline uint8_t bnxt_re_set_hdr_flags(struct bnxt_re_bsqe *hdr, - uint32_t send_flags, uint8_t sqsig) +static int bnxt_re_calc_inline_len(struct ibv_send_wr *swr, uint32_t max_ils) +{ + int illen, indx; + + illen = 0; + for (indx = 0; indx < swr->num_sge; indx++) + illen += swr->sg_list[indx].length; + if (illen > max_ils) + illen = max_ils; + return illen; +} + +static int bnxt_re_calc_posted_wqe_slots(struct bnxt_re_queue *que, void *wr, + uint32_t max_ils, bool is_rq) +{ + struct ibv_send_wr *swr; + struct ibv_recv_wr *rwr; + uint32_t wqe_byte; + uint32_t nsge; + int ilsize; + + swr = wr; + rwr = wr; + + nsge = is_rq ? rwr->num_sge : swr->num_sge; + wqe_byte = bnxt_re_calc_wqe_sz(nsge); + if (!is_rq && (swr->send_flags & IBV_SEND_INLINE)) { + ilsize = bnxt_re_calc_inline_len(swr, max_ils); + wqe_byte = get_aligned(ilsize, sizeof(struct bnxt_re_sge)); + wqe_byte += sizeof(struct bnxt_re_bsqe); + } + + return (wqe_byte / que->stride); +} + +static inline bool bnxt_re_set_hdr_flags(struct bnxt_re_bsqe *hdr, + uint32_t send_flags, uint8_t sqsig, + uint32_t slots) { uint8_t is_inline = false; uint32_t hdrval = 0; @@ -1117,36 +1241,38 @@ static inline uint8_t bnxt_re_set_hdr_flags(struct bnxt_re_bsqe *hdr, << BNXT_RE_HDR_FLAGS_SHIFT); is_inline = true; } + hdrval |= (slots & BNXT_RE_HDR_WS_MASK) << BNXT_RE_HDR_WS_SHIFT; hdr->rsv_ws_fl_wt = htole32(hdrval); return is_inline; } -static int bnxt_re_build_sge(struct bnxt_re_sge *sge, struct ibv_sge *sg_list, - uint32_t num_sge, uint8_t is_inline) { +static int bnxt_re_build_sge(struct bnxt_re_queue *que, struct ibv_sge *sg_list, + uint32_t num_sge, uint8_t is_inline, + uint32_t *idx) +{ + struct bnxt_re_sge *sge; int indx, length = 0; void *dst; - if (!num_sge) { - memset(sge, 0, sizeof(*sge)); + if (!num_sge) return 0; - } if (is_inline) { - dst = sge; for (indx = 0; indx < num_sge; indx++) { + dst = bnxt_re_get_hwqe(que, *idx); + (*idx)++; length += sg_list[indx].length; - if (length > BNXT_RE_MAX_INLINE_SIZE) - return -ENOMEM; memcpy(dst, (void *)(uintptr_t)sg_list[indx].addr, sg_list[indx].length); - dst = dst + sg_list[indx].length; } } else { for (indx = 0; indx < num_sge; indx++) { - sge[indx].pa = htole64(sg_list[indx].addr); - sge[indx].lkey = htole32(sg_list[indx].lkey); - sge[indx].length = htole32(sg_list[indx].length); + sge = bnxt_re_get_hwqe(que, *idx); + (*idx)++; + sge->pa = htole64(sg_list[indx].addr); + sge->lkey = htole32(sg_list[indx].lkey); + sge->length = htole32(sg_list[indx].length); length += sg_list[indx].length; } } @@ -1164,6 +1290,7 @@ static void bnxt_re_fill_psns(struct bnxt_re_qp *qp, struct bnxt_re_wrid *wrid, psns = wrid->psns; psns_ext = wrid->psns_ext; + len = wrid->bytes; if (qp->qptyp == IBV_QPT_RC) { opc_spsn = qp->sq_psn & BNXT_RE_PSNS_SPSN_MASK; @@ -1183,7 +1310,7 @@ static void bnxt_re_fill_psns(struct bnxt_re_qp *qp, struct bnxt_re_wrid *wrid, psns->opc_spsn = htole32(opc_spsn); psns->flg_npsn = htole32(flg_npsn); if (bnxt_re_is_chip_gen_p5(qp->cctx)) - psns_ext->st_slot_idx = 0; + psns_ext->st_slot_idx = wrid->st_slot_idx; } static void bnxt_re_fill_wrid(struct bnxt_re_wrid *wrid, uint64_t wr_id, @@ -1199,16 +1326,19 @@ static void bnxt_re_fill_wrid(struct bnxt_re_wrid *wrid, uint64_t wr_id, wrid->slots = slots; } -static int bnxt_re_build_send_sqe(struct bnxt_re_qp *qp, void *wqe, - struct ibv_send_wr *wr, uint8_t is_inline) +static int bnxt_re_build_send_sqe(struct bnxt_re_qp *qp, + struct ibv_send_wr *wr, + struct bnxt_re_bsqe *hdr, + uint8_t is_inline, uint32_t *idx) { - struct bnxt_re_sge *sge = ((void *)wqe + bnxt_re_get_sqe_hdr_sz()); - struct bnxt_re_bsqe *hdr = wqe; - uint32_t wrlen, hdrval = 0; - uint8_t opcode, qesize; + struct bnxt_re_queue *que; + uint32_t hdrval = 0; + uint8_t opcode; int len; - len = bnxt_re_build_sge(sge, wr->sg_list, wr->num_sge, is_inline); + que = qp->jsqq->hwque; + len = bnxt_re_build_sge(que, wr->sg_list, wr->num_sge, + is_inline, idx); if (len < 0) return len; hdr->lhdr.qkey_len = htole64((uint64_t)len); @@ -1218,34 +1348,22 @@ static int bnxt_re_build_send_sqe(struct bnxt_re_qp *qp, void *wqe, if (opcode == BNXT_RE_WR_OPCD_INVAL) return -EINVAL; hdrval = (opcode & BNXT_RE_HDR_WT_MASK); - - if (is_inline) { - wrlen = get_aligned(len, 16); - qesize = wrlen >> 4; - } else { - qesize = wr->num_sge; - } - /* HW requires wqe size has room for atleast one sge even if none was - * supplied by application - */ - if (!wr->num_sge) - qesize++; - qesize += (bnxt_re_get_sqe_hdr_sz() >> 4); - hdrval |= (qesize & BNXT_RE_HDR_WS_MASK) << BNXT_RE_HDR_WS_SHIFT; hdr->rsv_ws_fl_wt |= htole32(hdrval); return len; } -static int bnxt_re_build_ud_sqe(struct bnxt_re_qp *qp, void *wqe, - struct ibv_send_wr *wr, uint8_t is_inline) +static int bnxt_re_build_ud_sqe(struct bnxt_re_qp *qp, struct ibv_send_wr *wr, + struct bnxt_re_bsqe *hdr, uint8_t is_inline, + uint32_t *idx) { - struct bnxt_re_send *sqe = ((void *)wqe + sizeof(struct bnxt_re_bsqe)); - struct bnxt_re_bsqe *hdr = wqe; + struct bnxt_re_send *sqe; struct bnxt_re_ah *ah; uint64_t qkey; int len; - len = bnxt_re_build_send_sqe(qp, wqe, wr, is_inline); + sqe = bnxt_re_get_hwqe(qp->jsqq->hwque, *idx); + (*idx)++; + len = bnxt_re_build_send_sqe(qp, wr, hdr, is_inline, idx); if (!wr->wr.ud.ah) { len = -EINVAL; goto bail; @@ -1259,28 +1377,33 @@ bail: return len; } -static int bnxt_re_build_rdma_sqe(struct bnxt_re_qp *qp, void *wqe, - struct ibv_send_wr *wr, uint8_t is_inline) +static int bnxt_re_build_rdma_sqe(struct bnxt_re_qp *qp, + struct bnxt_re_bsqe *hdr, + struct ibv_send_wr *wr, + uint8_t is_inline, uint32_t *idx) { - struct bnxt_re_rdma *sqe = ((void *)wqe + sizeof(struct bnxt_re_bsqe)); + struct bnxt_re_rdma *sqe; int len; - len = bnxt_re_build_send_sqe(qp, wqe, wr, is_inline); + sqe = bnxt_re_get_hwqe(qp->jsqq->hwque, *idx); + (*idx)++; + len = bnxt_re_build_send_sqe(qp, wr, hdr, is_inline, idx); sqe->rva = htole64(wr->wr.rdma.remote_addr); sqe->rkey = htole32(wr->wr.rdma.rkey); return len; } -static int bnxt_re_build_cns_sqe(struct bnxt_re_qp *qp, void *wqe, - struct ibv_send_wr *wr) +static int bnxt_re_build_cns_sqe(struct bnxt_re_qp *qp, + struct bnxt_re_bsqe *hdr, + struct ibv_send_wr *wr, uint32_t *idx) { - struct bnxt_re_bsqe *hdr = wqe; - struct bnxt_re_atomic *sqe = ((void *)wqe + - sizeof(struct bnxt_re_bsqe)); + struct bnxt_re_atomic *sqe; int len; - len = bnxt_re_build_send_sqe(qp, wqe, wr, false); + sqe = bnxt_re_get_hwqe(qp->jsqq->hwque, *idx); + (*idx)++; + len = bnxt_re_build_send_sqe(qp, wr, hdr, false, idx); hdr->key_immd = htole32(wr->wr.atomic.rkey); hdr->lhdr.rva = htole64(wr->wr.atomic.remote_addr); sqe->cmp_dt = htole64(wr->wr.atomic.compare_add); @@ -1289,15 +1412,16 @@ static int bnxt_re_build_cns_sqe(struct bnxt_re_qp *qp, void *wqe, return len; } -static int bnxt_re_build_fna_sqe(struct bnxt_re_qp *qp, void *wqe, - struct ibv_send_wr *wr) +static int bnxt_re_build_fna_sqe(struct bnxt_re_qp *qp, + struct bnxt_re_bsqe *hdr, + struct ibv_send_wr *wr, uint32_t *idx) { - struct bnxt_re_bsqe *hdr = wqe; - struct bnxt_re_atomic *sqe = ((void *)wqe + - sizeof(struct bnxt_re_bsqe)); + struct bnxt_re_atomic *sqe; int len; - len = bnxt_re_build_send_sqe(qp, wqe, wr, false); + sqe = bnxt_re_get_hwqe(qp->jsqq->hwque, *idx); + (*idx)++; + len = bnxt_re_build_send_sqe(qp, wr, hdr, false, idx); hdr->key_immd = htole32(wr->wr.atomic.rkey); hdr->lhdr.rva = htole64(wr->wr.atomic.remote_addr); sqe->cmp_dt = htole64(wr->wr.atomic.compare_add); @@ -1311,13 +1435,16 @@ int bnxt_re_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, struct bnxt_re_qp *qp = to_bnxt_re_qp(ibvqp); struct bnxt_re_queue *sq = qp->jsqq->hwque; struct bnxt_re_wrid *wrid; + struct bnxt_re_send *sqe; uint8_t is_inline = false; struct bnxt_re_bsqe *hdr; + uint32_t swq_idx, slots; int ret = 0, bytes = 0; bool ring_db = false; - uint32_t swq_idx; - uint32_t sig; - void *sqe; + uint32_t wqe_size; + uint32_t max_ils; + uint8_t sig = 0; + uint32_t idx; pthread_spin_lock(&sq->qlock); while (wr) { @@ -1335,18 +1462,20 @@ int bnxt_re_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, goto bad_wr; } - if (bnxt_re_is_que_full(sq) || + max_ils = qp->cap.max_inline; + wqe_size = bnxt_re_calc_posted_wqe_slots(sq, wr, max_ils, false); + slots = (qp->qpmode == BNXT_RE_WQE_MODE_STATIC) ? 8 : wqe_size; + if (bnxt_re_is_que_full(sq, slots) || wr->num_sge > qp->cap.max_ssge) { *bad = wr; ret = ENOMEM; goto bad_wr; } - sqe = (void *)(sq->va + (sq->tail * sq->stride)); - memset(sqe, 0, bnxt_re_get_sqe_sz()); - hdr = sqe; + idx = 0; + hdr = bnxt_re_get_hwqe(sq, idx++); is_inline = bnxt_re_set_hdr_flags(hdr, wr->send_flags, - qp->cap.sqsig); + qp->cap.sqsig, wqe_size); switch (wr->opcode) { case IBV_WR_SEND_WITH_IMM: /* Since our h/w is LE and user supplies raw-data in @@ -1357,27 +1486,31 @@ int bnxt_re_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, hdr->key_immd = htole32(be32toh(wr->imm_data)); SWITCH_FALLTHROUGH; case IBV_WR_SEND: - if (qp->qptyp == IBV_QPT_UD) - bytes = bnxt_re_build_ud_sqe(qp, sqe, wr, - is_inline); - else - bytes = bnxt_re_build_send_sqe(qp, sqe, wr, - is_inline); + if (qp->qptyp == IBV_QPT_UD) { + bytes = bnxt_re_build_ud_sqe(qp, wr, hdr, + is_inline, &idx); + } else { + sqe = bnxt_re_get_hwqe(sq, idx++); + memset(sqe, 0, sizeof(struct bnxt_re_send)); + bytes = bnxt_re_build_send_sqe(qp, wr, hdr, + is_inline, + &idx); + } break; case IBV_WR_RDMA_WRITE_WITH_IMM: hdr->key_immd = htole32(be32toh(wr->imm_data)); SWITCH_FALLTHROUGH; case IBV_WR_RDMA_WRITE: - bytes = bnxt_re_build_rdma_sqe(qp, sqe, wr, is_inline); + bytes = bnxt_re_build_rdma_sqe(qp, hdr, wr, is_inline, &idx); break; case IBV_WR_RDMA_READ: - bytes = bnxt_re_build_rdma_sqe(qp, sqe, wr, false); + bytes = bnxt_re_build_rdma_sqe(qp, hdr, wr, false, &idx); break; case IBV_WR_ATOMIC_CMP_AND_SWP: - bytes = bnxt_re_build_cns_sqe(qp, sqe, wr); + bytes = bnxt_re_build_cns_sqe(qp, hdr, wr, &idx); break; case IBV_WR_ATOMIC_FETCH_AND_ADD: - bytes = bnxt_re_build_fna_sqe(qp, sqe, wr); + bytes = bnxt_re_build_fna_sqe(qp, hdr, wr, &idx); break; default: bytes = -EINVAL; @@ -1392,10 +1525,11 @@ int bnxt_re_post_send(struct ibv_qp *ibvqp, struct ibv_send_wr *wr, wrid = bnxt_re_get_swqe(qp->jsqq, &swq_idx); sig = ((wr->send_flags & IBV_SEND_SIGNALED) || qp->cap.sqsig); - bnxt_re_fill_wrid(wrid, wr->wr_id, bytes, sig, sq->tail, 1); + bnxt_re_fill_wrid(wrid, wr->wr_id, bytes, + sig, sq->tail, slots); bnxt_re_fill_psns(qp, wrid, wr->opcode, bytes); bnxt_re_jqq_mod_start(qp->jsqq, swq_idx); - bnxt_re_incr_tail(sq, 1); + bnxt_re_incr_tail(sq, slots); qp->wqe_cnt++; wr = wr->next; ring_db = true; @@ -1421,17 +1555,14 @@ bad_wr: return ret; } -static int bnxt_re_build_rqe(struct bnxt_re_qp *qp, struct ibv_recv_wr *wr, - void *rqe, uint32_t idx) +static int bnxt_re_build_rqe(struct bnxt_re_queue *rq, struct ibv_recv_wr *wr, + struct bnxt_re_brqe *hdr, uint32_t wqe_sz, + uint32_t *idx, uint32_t wqe_idx) { - struct bnxt_re_brqe *hdr = rqe; - struct bnxt_re_sge *sge; - int wqe_sz, len; uint32_t hdrval; + int len; - sge = (rqe + bnxt_re_get_rqe_hdr_sz()); - - len = bnxt_re_build_sge(sge, wr->sg_list, wr->num_sge, false); + len = bnxt_re_build_sge(rq, wr->sg_list, wr->num_sge, false, idx); wqe_sz = wr->num_sge + (bnxt_re_get_rqe_hdr_sz() >> 4); /* 16B align */ /* HW requires wqe size has room for atleast one sge even if none was * supplied by application @@ -1441,7 +1572,7 @@ static int bnxt_re_build_rqe(struct bnxt_re_qp *qp, struct ibv_recv_wr *wr, hdrval = BNXT_RE_WR_OPCD_RECV; hdrval |= ((wqe_sz & BNXT_RE_HDR_WS_MASK) << BNXT_RE_HDR_WS_SHIFT); hdr->rsv_ws_fl_wt = htole32(hdrval); - hdr->wrid = htole32(idx); + hdr->wrid = htole32(wqe_idx); return len; } @@ -1452,8 +1583,11 @@ int bnxt_re_post_recv(struct ibv_qp *ibvqp, struct ibv_recv_wr *wr, struct bnxt_re_qp *qp = to_bnxt_re_qp(ibvqp); struct bnxt_re_queue *rq = qp->jrqq->hwque; struct bnxt_re_wrid *swque; - uint32_t swq_idx; - void *rqe; + struct bnxt_re_brqe *hdr; + struct bnxt_re_rqe *rqe; + uint32_t slots, swq_idx; + uint32_t wqe_size; + uint32_t idx = 0; int ret; pthread_spin_lock(&rq->qlock); @@ -1465,17 +1599,24 @@ int bnxt_re_post_recv(struct ibv_qp *ibvqp, struct ibv_recv_wr *wr, return EINVAL; } - if (bnxt_re_is_que_full(rq) || + wqe_size = bnxt_re_calc_posted_wqe_slots(rq, wr, 0, true); + slots = rq->max_slots; + if (bnxt_re_is_que_full(rq, slots) || wr->num_sge > qp->cap.max_rsge) { pthread_spin_unlock(&rq->qlock); *bad = wr; return ENOMEM; } - rqe = (void *)(rq->va + (rq->tail * rq->stride)); - memset(rqe, 0, bnxt_re_get_rqe_sz()); + idx = 0; swque = bnxt_re_get_swqe(qp->jrqq, &swq_idx); - ret = bnxt_re_build_rqe(qp, wr, rqe, swq_idx); + hdr = bnxt_re_get_hwqe(rq, idx++); + /* Just to build clean rqe */ + rqe = bnxt_re_get_hwqe(rq, idx++); + memset(rqe, 0, sizeof(struct bnxt_re_rqe)); + /* Fill SGEs */ + + ret = bnxt_re_build_rqe(rq, wr, hdr, wqe_size, &idx, swq_idx); if (ret < 0) { pthread_spin_unlock(&rq->qlock); *bad = wr; @@ -1483,9 +1624,9 @@ int bnxt_re_post_recv(struct ibv_qp *ibvqp, struct ibv_recv_wr *wr, } swque = bnxt_re_get_swqe(qp->jrqq, NULL); - bnxt_re_fill_wrid(swque, wr->wr_id, ret, 0, rq->tail, 1); + bnxt_re_fill_wrid(swque, wr->wr_id, ret, 0, rq->tail, slots); bnxt_re_jqq_mod_start(qp->jrqq, swq_idx); - bnxt_re_incr_tail(rq, 1); + bnxt_re_incr_tail(rq, slots); wr = wr->next; bnxt_re_ring_rq_db(qp); } @@ -1644,12 +1785,20 @@ static int bnxt_re_build_srqe(struct bnxt_re_srq *srq, struct bnxt_re_wrid *wrid; int wqe_sz, len, next; uint32_t hdrval = 0; + int indx; sge = (srqe + bnxt_re_get_srqe_hdr_sz()); next = srq->start_idx; wrid = &srq->srwrid[next]; - len = bnxt_re_build_sge(sge, wr->sg_list, wr->num_sge, false); + len = 0; + for (indx = 0; indx < wr->num_sge; indx++, sge++) { + sge->pa = htole64(wr->sg_list[indx].addr); + sge->lkey = htole32(wr->sg_list[indx].lkey); + sge->length = htole32(wr->sg_list[indx].length); + len += wr->sg_list[indx].length; + } + hdrval = BNXT_RE_WR_OPCD_RECV; wqe_sz = wr->num_sge + (bnxt_re_get_srqe_hdr_sz() >> 4); /* 16B align */ hdrval |= ((wqe_sz & BNXT_RE_HDR_WS_MASK) << BNXT_RE_HDR_WS_SHIFT);