From patchwork Wed Nov 8 09:57:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sagi Grimberg X-Patchwork-Id: 10048161 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 4156A60247 for ; Wed, 8 Nov 2017 09:58:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 289C1292D6 for ; Wed, 8 Nov 2017 09:58:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1CA3E2A4FE; Wed, 8 Nov 2017 09:58:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.1 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, URIBL_BLACK autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 98EBF292D6 for ; Wed, 8 Nov 2017 09:58:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751826AbdKHJ6S (ORCPT ); Wed, 8 Nov 2017 04:58:18 -0500 Received: from bombadil.infradead.org ([65.50.211.133]:59641 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751786AbdKHJ6R (ORCPT ); Wed, 8 Nov 2017 04:58:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=7sTlN1fgpuz5FpGvvvrGkqaJYw9N/nt+KAW41HMsNvI=; b=dyCt3obUyQBp0hOk+AUWTgkH3 apULjTRLjOL0TL/wddUZYWp3HqAeql1/U8mHa1tppJXXbIFhfPPhjzFc/f80ub9I8JdKN36PCxZEy 0gHv8UXjqDltgqPybWyogzPNDlg+0c2p2A62Aq6+3drv3ji5N4qFJvBy3+Noq4LJSluGr4tqga+yc AgrIqKihss2uZNDRj20VhAHwBQqIKe3wEf6i1SYxjl1IHoROc/y+FP3UUEV1KnDrA/Cm/Pg13bBQh 3Ak38QkSHLWPeIWT1o1yS09fuELJ20tCXLWllFBqZGCo2d0AysI/biTyfJnq4vEGHeCr0YpUFVFKL /toW3jhUQ==; Received: from [31.154.58.122] (helo=bombadil.infradead.org) by bombadil.infradead.org with esmtpsa (Exim 4.87 #1 (Red Hat Linux)) id 1eCN7e-0001zj-51; Wed, 08 Nov 2017 09:58:14 +0000 From: Sagi Grimberg To: linux-rdma@vger.kernel.org Cc: linux-nvme@lists.infradead.org, Christoph Hellwig , Max Gurtuvoy Subject: [PATCH v3 4/9] IB/srpt: use implicit CQ allocation Date: Wed, 8 Nov 2017 11:57:37 +0200 Message-Id: <20171108095742.25365-5-sagi@grimberg.me> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20171108095742.25365-1-sagi@grimberg.me> References: <20171108095742.25365-1-sagi@grimberg.me> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Sagi Grimberg [hch: ported to the new API] Signed-off-by: Christoph Hellwig --- drivers/infiniband/ulp/srpt/ib_srpt.c | 46 ++++++++++++----------------------- drivers/infiniband/ulp/srpt/ib_srpt.h | 1 - 2 files changed, 15 insertions(+), 32 deletions(-) diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c index 9e8e9220f816..256d0d5b32e5 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.c +++ b/drivers/infiniband/ulp/srpt/ib_srpt.c @@ -798,7 +798,7 @@ static int srpt_zerolength_write(struct srpt_rdma_ch *ch) static void srpt_zerolength_write_done(struct ib_cq *cq, struct ib_wc *wc) { - struct srpt_rdma_ch *ch = cq->cq_context; + struct srpt_rdma_ch *ch = wc->qp->qp_context; if (wc->status == IB_WC_SUCCESS) { srpt_process_wait_list(ch); @@ -1201,7 +1201,7 @@ static int srpt_abort_cmd(struct srpt_send_ioctx *ioctx) */ static void srpt_rdma_read_done(struct ib_cq *cq, struct ib_wc *wc) { - struct srpt_rdma_ch *ch = cq->cq_context; + struct srpt_rdma_ch *ch = wc->qp->qp_context; struct srpt_send_ioctx *ioctx = container_of(wc->wr_cqe, struct srpt_send_ioctx, rdma_cqe); @@ -1526,7 +1526,7 @@ static void srpt_handle_new_iu(struct srpt_rdma_ch *ch, static void srpt_recv_done(struct ib_cq *cq, struct ib_wc *wc) { - struct srpt_rdma_ch *ch = cq->cq_context; + struct srpt_rdma_ch *ch = wc->qp->qp_context; struct srpt_recv_ioctx *ioctx = container_of(wc->wr_cqe, struct srpt_recv_ioctx, ioctx.cqe); @@ -1580,7 +1580,7 @@ static void srpt_process_wait_list(struct srpt_rdma_ch *ch) */ static void srpt_send_done(struct ib_cq *cq, struct ib_wc *wc) { - struct srpt_rdma_ch *ch = cq->cq_context; + struct srpt_rdma_ch *ch = wc->qp->qp_context; struct srpt_send_ioctx *ioctx = container_of(wc->wr_cqe, struct srpt_send_ioctx, ioctx.cqe); enum srpt_command_state state; @@ -1626,23 +1626,14 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch) goto out; retry: - ch->cq = ib_alloc_cq(sdev->device, ch, ch->rq_size + srp_sq_size, - 0 /* XXX: spread CQs */, IB_POLL_WORKQUEUE); - if (IS_ERR(ch->cq)) { - ret = PTR_ERR(ch->cq); - pr_err("failed to create CQ cqe= %d ret= %d\n", - ch->rq_size + srp_sq_size, ret); - goto out; - } - + qp_init->create_flags = IB_QP_CREATE_ASSIGN_CQS; qp_init->qp_context = (void *)ch; qp_init->event_handler = (void(*)(struct ib_event *, void*))srpt_qp_event; - qp_init->send_cq = ch->cq; - qp_init->recv_cq = ch->cq; qp_init->srq = sdev->srq; qp_init->sq_sig_type = IB_SIGNAL_REQ_WR; qp_init->qp_type = IB_QPT_RC; + qp_init->poll_ctx = IB_POLL_WORKQUEUE; /* * We divide up our send queue size into half SEND WRs to send the * completions, and half R/W contexts to actually do the RDMA @@ -1653,6 +1644,9 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch) qp_init->cap.max_send_wr = srp_sq_size / 2; qp_init->cap.max_rdma_ctxs = srp_sq_size / 2; qp_init->cap.max_send_sge = min(attrs->max_sge, SRPT_MAX_SG_PER_WQE); + + qp_init->cap.max_recv_wr = ch->rq_size; + qp_init->port_num = ch->sport->port; ch->qp = ib_create_qp(sdev->pd, qp_init); @@ -1660,19 +1654,17 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch) ret = PTR_ERR(ch->qp); if (ret == -ENOMEM) { srp_sq_size /= 2; - if (srp_sq_size >= MIN_SRPT_SQ_SIZE) { - ib_destroy_cq(ch->cq); + if (srp_sq_size >= MIN_SRPT_SQ_SIZE) goto retry; - } } pr_err("failed to create_qp ret= %d\n", ret); - goto err_destroy_cq; + goto out; } atomic_set(&ch->sq_wr_avail, qp_init->cap.max_send_wr); - pr_debug("%s: max_cqe= %d max_sge= %d sq_size = %d cm_id= %p\n", - __func__, ch->cq->cqe, qp_init->cap.max_send_sge, + pr_debug("%s: max_sge= %d sq_size = %d cm_id= %p\n", + __func__, qp_init->cap.max_send_sge, qp_init->cap.max_send_wr, ch->cm_id); ret = srpt_init_ch_qp(ch, ch->qp); @@ -1685,17 +1677,9 @@ static int srpt_create_ch_ib(struct srpt_rdma_ch *ch) err_destroy_qp: ib_destroy_qp(ch->qp); -err_destroy_cq: - ib_free_cq(ch->cq); goto out; } -static void srpt_destroy_ch_ib(struct srpt_rdma_ch *ch) -{ - ib_destroy_qp(ch->qp); - ib_free_cq(ch->cq); -} - /** * srpt_close_ch() - Close an RDMA channel. * @@ -1812,7 +1796,7 @@ static void srpt_release_channel_work(struct work_struct *w) ib_destroy_cm_id(ch->cm_id); - srpt_destroy_ch_ib(ch); + ib_destroy_qp(ch->qp); srpt_free_ioctx_ring((struct srpt_ioctx **)ch->ioctx_ring, ch->sport->sdev, ch->rq_size, @@ -2070,7 +2054,7 @@ static int srpt_cm_req_recv(struct ib_cm_id *cm_id, ch->sess = NULL; destroy_ib: - srpt_destroy_ch_ib(ch); + ib_destroy_qp(ch->qp); free_ring: srpt_free_ioctx_ring((struct srpt_ioctx **)ch->ioctx_ring, diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.h b/drivers/infiniband/ulp/srpt/ib_srpt.h index 1b817e51b84b..4ab0d94af174 100644 --- a/drivers/infiniband/ulp/srpt/ib_srpt.h +++ b/drivers/infiniband/ulp/srpt/ib_srpt.h @@ -265,7 +265,6 @@ enum rdma_ch_state { struct srpt_rdma_ch { struct ib_cm_id *cm_id; struct ib_qp *qp; - struct ib_cq *cq; struct ib_cqe zw_cqe; struct kref kref; int rq_size;