From patchwork Fri Feb 10 23:56:11 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 9567625 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id ACAA7602B6 for ; Fri, 10 Feb 2017 23:56:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 98FE9285D2 for ; Fri, 10 Feb 2017 23:56:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8DB54285EA; Fri, 10 Feb 2017 23:56:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 26EE4285D2 for ; Fri, 10 Feb 2017 23:56:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751665AbdBJX4Z (ORCPT ); Fri, 10 Feb 2017 18:56:25 -0500 Received: from esa1.hgst.iphmx.com ([68.232.141.245]:7613 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751584AbdBJX4W (ORCPT ); Fri, 10 Feb 2017 18:56:22 -0500 X-IronPort-AV: E=Sophos;i="5.33,348,1477929600"; d="scan'208";a="83251618" Received: from unknown (HELO milsmgep15.sandisk.com) ([63.163.107.21]) by ob1.hgst.iphmx.com with ESMTP; 11 Feb 2017 07:56:19 +0800 Received: from MILHUBIP03.sdcorp.global.sandisk.com (Unknown_Domain [10.201.67.162]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by (Symantec Messaging Gateway) with SMTP id 5C.5A.57638.3235E985; Fri, 10 Feb 2017 15:56:19 -0800 (PST) Received: from milsmgip11.sandisk.com (10.177.9.6) by MILHUBIP03.sdcorp.global.sandisk.com (10.177.9.96) with Microsoft SMTP Server id 14.3.319.2; Fri, 10 Feb 2017 15:56:14 -0800 X-AuditID: 0ac94369-26dee9800001e126-53-589e5323e815 Received: from exp-402881.sdcorp.global.sandisk.com ( [10.177.8.100]) by (Symantec Messaging Gateway) with SMTP id 40.B4.11415.F135E985; Fri, 10 Feb 2017 15:56:15 -0800 (PST) From: Bart Van Assche To: Doug Ledford CC: , Bart Van Assche , Christoph Hellwig , Israel Rukshin , Max Gurtovoy , Laurence Oberman Subject: [PATCH 8/8] IB/srp: Drain the send queue before destroying a QP Date: Fri, 10 Feb 2017 15:56:11 -0800 Message-ID: <20170210235611.3243-9-bart.vanassche@sandisk.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170210235611.3243-1-bart.vanassche@sandisk.com> References: <20170210235611.3243-1-bart.vanassche@sandisk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrILMWRmVeSWpSXmKPExsXCddJ5ka5y8LwIg85OKYuX5z+wWqxcfZTJ 4sbitSwWzw71slic//CKyeL0owPsDmweu282sHk8m36YyeP9vqtsHp83yQWwRHHZpKTmZJal FunbJXBlXPndx15wWKzi0IlT7A2M24W6GDk5JARMJD41X2buYuTiEBJYyiTx/vhHNghnG6PE g1mPGGGqfrReY4JIbGaUWHDzCzNIgk3ASOLb+5ksILaIgJrEpleL2EGKmAWeMko8Ot7MBpIQ FvCQ6Hh3CcxmEVCVOLPpK5DNwcErYCcxeWcFxAJ5iV1tF1lBbE4Be4nvy6aB2UJAJW++fmIF mSkhMIlV4sy/RWCLeQUEJU7OfAK2mFlAQuLgixfMEA3qEieXzGeawCg0C0nZLCRlCxiZVjGK 5WbmFOempxYYmuoVJ+alZBZn6yXn525ihAR85g7Gu0+8DzEKcDAq8fAmHJ8bIcSaWFZcmXuI UYKDWUmEN8l1XoQQb0piZVVqUX58UWlOavEhRmkOFiVx3nMyUyOEBNITS1KzU1MLUotgskwc nFINjIGMHJ7/lA5rflmtzLXWestGm9jvVfUsTYeSxA0fbVBnEclatIRHYP6kQC+ty5zRKi+u Lnq/ca9gNLPOpkfON9mFvsZUysa5B3b/U6vLFW7Q3T/57LebNstOiwYXaky41r79qirb55b8 CXb1tzjUK+M614nu6Ap/KrCzad+14pXWs9i5M5u2KbEUZyQaajEXFScCAFQg4VV0AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrIJMWRmVeSWpSXmKPExsXCtZEjRVc+eF6EwdNPrBYHf7YxWrw8/4HV YuXqo0wWNxavZbF4dqiXxeL8h1dMFqcfHWB3YPfYfbOBzePZ9MNMHu/3XWXzmLbmPJPH501y AaxRXDYpqTmZZalF+nYJXBlXfvexFxwWqzh04hR7A+N2oS5GTg4JAROJH63XmLoYuTiEBDYy SjR297KAJNgEjCS+vZ8JZosIqElserWIHaSIWeA5o8TF/ceZQRLCAh4SHe8usYHYLAKqEmc2 fQWzeQXsJKbNameF2CAvsavtIpjNKWAv8X3ZNDBbCKjmzddPrBMYuRcwMqxiFMvNzCnOTc8s MDTUK07MS8ksztZLzs/dxAgOFM7IHYxPJ5ofYmTi4JRqYJSuWV/35m+wQ93Ob+r3FXrPNT+Z KWX7VMl04huxai6XM5yrC6x1dbc1pF2WnCIs6JTL+H8XT+h6iQtWji9TpF4ee/pSofJ4zBXJ aRfyH5oFHHauM999ZkJMBUfqVZ94Ff7rjBMq+6MLzbZ+9ju/9kwQe1TnCs+H+7pNuRmDGCpa LFl+WxxYr8RSnJFoqMVcVJwIAKgAB0/EAQAA MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A quote from the IB spec: However, if the Consumer does not wait for the Affiliated Asynchronous Last WQE Reached Event, then WQE and Data Segment leakage may occur. Therefore, it is good programming practice to tear down a QP that is associated with an SRQ by using the following process: * Put the QP in the Error State; * wait for the Affiliated Asynchronous Last WQE Reached Event; * either: * drain the CQ by invoking the Poll CQ verb and either wait for CQ to be empty or the number of Poll CQ operations has exceeded CQ capacity size; or * post another WR that completes on the same CQ and wait for this WR to return as a WC; * and then invoke a Destroy QP or Reset QP. Signed-off-by: Bart Van Assche Cc: Christoph Hellwig Cc: Israel Rukshin Cc: Max Gurtovoy Cc: Laurence Oberman Tested-by: Laurence Oberman Tested-by: Laurence Oberman --- drivers/infiniband/ulp/srp/ib_srp.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index 2f85255d2aca..b50733910f7e 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -471,9 +471,13 @@ static struct srp_fr_pool *srp_alloc_fr_pool(struct srp_target_port *target) * completion handler can access the queue pair while it is * being destroyed. */ -static void srp_destroy_qp(struct ib_qp *qp) +static void srp_destroy_qp(struct srp_rdma_ch *ch, struct ib_qp *qp) { - ib_drain_rq(qp); + spin_lock_irq(&ch->lock); + ib_process_cq_direct(ch->send_cq, -1); + spin_unlock_irq(&ch->lock); + + ib_drain_qp(qp); ib_destroy_qp(qp); } @@ -547,7 +551,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch) } if (ch->qp) - srp_destroy_qp(ch->qp); + srp_destroy_qp(ch, ch->qp); if (ch->recv_cq) ib_free_cq(ch->recv_cq); if (ch->send_cq) @@ -571,7 +575,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch) return 0; err_qp: - srp_destroy_qp(qp); + srp_destroy_qp(ch, qp); err_send_cq: ib_free_cq(send_cq); @@ -614,7 +618,7 @@ static void srp_free_ch_ib(struct srp_target_port *target, ib_destroy_fmr_pool(ch->fmr_pool); } - srp_destroy_qp(ch->qp); + srp_destroy_qp(ch, ch->qp); ib_free_cq(ch->send_cq); ib_free_cq(ch->recv_cq); @@ -1827,6 +1831,11 @@ static struct srp_iu *__srp_get_tx_iu(struct srp_rdma_ch *ch, return iu; } +/* + * Note: if this function is called from inside ib_drain_sq() then it will + * be called without ch->lock being held. If ib_drain_sq() dequeues a WQE + * with status IB_WC_SUCCESS then that's a bug. + */ static void srp_send_done(struct ib_cq *cq, struct ib_wc *wc) { struct srp_iu *iu = container_of(wc->wr_cqe, struct srp_iu, cqe);