From patchwork Tue Feb 14 18:56:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 9572527 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 3EAFA601E7 for ; Tue, 14 Feb 2017 18:56:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1726727DCD for ; Tue, 14 Feb 2017 18:56:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0C1602845E; Tue, 14 Feb 2017 18:56:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6FB972844A for ; Tue, 14 Feb 2017 18:56:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752868AbdBNS4v (ORCPT ); Tue, 14 Feb 2017 13:56:51 -0500 Received: from esa1.hgst.iphmx.com ([68.232.141.245]:26520 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752871AbdBNS4r (ORCPT ); Tue, 14 Feb 2017 13:56:47 -0500 X-IronPort-AV: E=Sophos;i="5.33,348,1477929600"; d="scan'208";a="86579476" Received: from unknown (HELO milsmgep14.sandisk.com) ([63.163.107.225]) by ob1.hgst.iphmx.com with ESMTP; 15 Feb 2017 02:56:44 +0800 Received: from MILHUBIP04.sdcorp.global.sandisk.com (Unknown_Domain [10.201.67.162]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by (Symantec Messaging Gateway) with SMTP id 68.9B.18895.BE253A85; Tue, 14 Feb 2017 10:56:44 -0800 (PST) Received: from milsmgip12.sandisk.com (10.177.9.6) by MILHUBIP04.sdcorp.global.sandisk.com (10.177.9.97) with Microsoft SMTP Server id 14.3.319.2; Tue, 14 Feb 2017 10:56:40 -0800 X-AuditID: 0ac94371-0fadb980000049cf-8b-58a352eb0128 Received: from exp-402881.sdcorp.global.sandisk.com ( [10.177.9.6]) by (Symantec Messaging Gateway) with SMTP id C3.D6.18148.8E253A85; Tue, 14 Feb 2017 10:56:40 -0800 (PST) From: Bart Van Assche To: Doug Ledford CC: , Bart Van Assche , Christoph Hellwig , Israel Rukshin , Max Gurtovoy , Laurence Oberman Subject: [PATCH v2 8/8] IB/srp: Drain the send queue before destroying a QP Date: Tue, 14 Feb 2017 10:56:36 -0800 Message-ID: <20170214185636.29250-9-bart.vanassche@sandisk.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170214185636.29250-1-bart.vanassche@sandisk.com> References: <20170214185636.29250-1-bart.vanassche@sandisk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrALMWRmVeSWpSXmKPExsXCddJ5ke6boMURBovusFq8PP+B1WLl6qNM FjcWr2WxeHaol8Xi/IdXTBanHx1gd2Dz2H2zgc3j2fTDTB7v911l8/i8SS6AJYrLJiU1J7Ms tUjfLoErY92veywFh8UqGia+Y25g3C7UxcjJISFgIrHv5QKmLkYuDiGBpUwSPQd+sYEkhAS2 MUrcXOgOU3Rs2wtGiKKNjBILpj5mAkmwCRhJfHs/kwXEFhFQk9j0ahE7SBGzwFNGiUfHm4Em cXAIC3hLnNxkAVLDIqAqcX/FXGYQm1fAXmLiil/sEAvkJXa1XWQFsTkFHCRmflvICnGEvcTZ /jlgMyUEZrFK7HjWC9UsKHFy5hOwxcwCEhIHX7xghmhQlzi5ZD7TBEahWUjKZiEpW8DItIpR LDczpzg3PbXA0ESvODEvJbM4Wy85P3cTIyTcC3cwvr7tfYhRgINRiYf3hOziCCHWxLLiytxD jBIczEoivBoOQCHelMTKqtSi/Pii0pzU4kOM0hwsSuK8WbJTI4QE0hNLUrNTUwtSi2CyTByc Ug2M1tIP18W9/mJwZlJb27fngbKLZF8+t96Ufj9qG0+da3ZjwvZPzzZe50t88fOess7MFen8 7+xFwmSmyM86yBPJbO+9fvkTmZuP3siafP8W9pIz/b+f22PmCTe8wsq3JSxdqeHF+146/WU5 S6LNq1eHlq04y7nte34B3+1752vuTPM4kD/zCOtxKyWW4oxEQy3mouJEANnV7J9zAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrAJMWRmVeSWpSXmKPExsXCtZGTTfdF0OIIg0lNghYHf7YxWrw8/4HV YuXqo0wWNxavZbF4dqiXxeL8h1dMFqcfHWB3YPfYfbOBzePZ9MNMHu/3XWXzmLbmPJPH501y AaxRXDYpqTmZZalF+nYJXBnrft1jKTgsVtEw8R1zA+N2oS5GTg4JAROJY9teMHYxcnEICaxn lDjy+gorSIJNwEji2/uZLCC2iICaxKZXi9hBipgFnjNKXNx/nLmLkYNDWMBb4uQmC5AaFgFV ifsr5jKD2LwC9hIL3/UxQiyQl9jVdhFsJqeAg8TMbwvBbCGgmrP9c9gnMHIvYGRYxSiWm5lT nJueWWBopFecmJeSWZytl5yfu4kREiZROxivTzQ/xMjEwSnVwLg1p/30jO7vs2K5+LVvum37 VWxTd0pJfX9mcbrym/0HPsooyuQb3W7X+Bc0YeNdsYWW/Iy6E/4ZSa38o6Qdf/fQ/b1sdfJe 7RJCP0R4+8/KqS6r/bJ8r4vp5Nux525au53b2CjlpPUytGXR1DOPer9JLXl0c7at7uxV1t8m irf/zD8n3dC3978SS3FGoqEWc1FxIgBStSX3wwEAAA== MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A quote from the IB spec: However, if the Consumer does not wait for the Affiliated Asynchronous Last WQE Reached Event, then WQE and Data Segment leakage may occur. Therefore, it is good programming practice to tear down a QP that is associated with an SRQ by using the following process: * Put the QP in the Error State; * wait for the Affiliated Asynchronous Last WQE Reached Event; * either: * drain the CQ by invoking the Poll CQ verb and either wait for CQ to be empty or the number of Poll CQ operations has exceeded CQ capacity size; or * post another WR that completes on the same CQ and wait for this WR to return as a WC; * and then invoke a Destroy QP or Reset QP. Signed-off-by: Bart Van Assche Cc: Christoph Hellwig Cc: Israel Rukshin Cc: Max Gurtovoy Cc: Laurence Oberman --- drivers/infiniband/ulp/srp/ib_srp.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c index b303b4331e98..39bc0f4e1303 100644 --- a/drivers/infiniband/ulp/srp/ib_srp.c +++ b/drivers/infiniband/ulp/srp/ib_srp.c @@ -466,9 +466,13 @@ static struct srp_fr_pool *srp_alloc_fr_pool(struct srp_target_port *target) * completion handler can access the queue pair while it is * being destroyed. */ -static void srp_destroy_qp(struct ib_qp *qp) +static void srp_destroy_qp(struct srp_rdma_ch *ch, struct ib_qp *qp) { - ib_drain_rq(qp); + spin_lock_irq(&ch->lock); + ib_process_cq_direct(ch->send_cq, -1); + spin_unlock_irq(&ch->lock); + + ib_drain_qp(qp); ib_destroy_qp(qp); } @@ -542,7 +546,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch) } if (ch->qp) - srp_destroy_qp(ch->qp); + srp_destroy_qp(ch, ch->qp); if (ch->recv_cq) ib_free_cq(ch->recv_cq); if (ch->send_cq) @@ -566,7 +570,7 @@ static int srp_create_ch_ib(struct srp_rdma_ch *ch) return 0; err_qp: - srp_destroy_qp(qp); + srp_destroy_qp(ch, qp); err_send_cq: ib_free_cq(send_cq); @@ -609,7 +613,7 @@ static void srp_free_ch_ib(struct srp_target_port *target, ib_destroy_fmr_pool(ch->fmr_pool); } - srp_destroy_qp(ch->qp); + srp_destroy_qp(ch, ch->qp); ib_free_cq(ch->send_cq); ib_free_cq(ch->recv_cq); @@ -1822,6 +1826,11 @@ static struct srp_iu *__srp_get_tx_iu(struct srp_rdma_ch *ch, return iu; } +/* + * Note: if this function is called from inside ib_drain_sq() then it will + * be called without ch->lock being held. If ib_drain_sq() dequeues a WQE + * with status IB_WC_SUCCESS then that's a bug. + */ static void srp_send_done(struct ib_cq *cq, struct ib_wc *wc) { struct srp_iu *iu = container_of(wc->wr_cqe, struct srp_iu, cqe);