From patchwork Tue Feb 14 18:56:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Bart Van Assche X-Patchwork-Id: 9572531 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 695D8601E7 for ; Tue, 14 Feb 2017 18:56:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 40EBC205A4 for ; Tue, 14 Feb 2017 18:56:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 32CC126E1A; Tue, 14 Feb 2017 18:56:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A066627DCD for ; Tue, 14 Feb 2017 18:56:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752324AbdBNS4x (ORCPT ); Tue, 14 Feb 2017 13:56:53 -0500 Received: from esa1.hgst.iphmx.com ([68.232.141.245]:26530 "EHLO esa1.hgst.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752861AbdBNS4s (ORCPT ); Tue, 14 Feb 2017 13:56:48 -0500 X-IronPort-AV: E=Sophos;i="5.33,348,1477929600"; d="scan'208";a="86579489" Received: from unknown (HELO milsmgep14.sandisk.com) ([63.163.107.225]) by ob1.hgst.iphmx.com with ESMTP; 15 Feb 2017 02:56:44 +0800 Received: from MILHUBIP04.sdcorp.global.sandisk.com (Unknown_Domain [10.201.67.162]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by (Symantec Messaging Gateway) with SMTP id D8.9B.18895.CE253A85; Tue, 14 Feb 2017 10:56:44 -0800 (PST) Received: from milsmgip12.sandisk.com (10.177.8.100) by MILHUBIP04.sdcorp.global.sandisk.com (10.177.9.97) with Microsoft SMTP Server id 14.3.319.2; Tue, 14 Feb 2017 10:56:40 -0800 X-AuditID: 0ac94371-0fadb980000049cf-8c-58a352ec3186 Received: from exp-402881.sdcorp.global.sandisk.com ( [10.177.9.6]) by (Symantec Messaging Gateway) with SMTP id 53.D6.18148.8E253A85; Tue, 14 Feb 2017 10:56:40 -0800 (PST) From: Bart Van Assche To: Doug Ledford CC: , Bart Van Assche , Steve Wise , Chuck Lever , Christoph Hellwig , Max Gurtovoy Subject: [PATCH v2 7/8] IB/core: Add support for draining IB_POLL_DIRECT completion queues Date: Tue, 14 Feb 2017 10:56:35 -0800 Message-ID: <20170214185636.29250-8-bart.vanassche@sandisk.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170214185636.29250-1-bart.vanassche@sandisk.com> References: <20170214185636.29250-1-bart.vanassche@sandisk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrNLMWRmVeSWpSXmKPExsXCddJ5ke6boMURBsvviFj8v/ucyeLl+Q+s FitXH2WyeHaol8Xi9KMD7Bb/1nazOLB57L7ZwObxbPphJo9Hz88xenx8eovF4/2+q2wenzfJ BbBFcdmkpOZklqUW6dslcGWcvreYsWCWcsW0t3sZGxiPynQxcnJICJhInNj7ia2LkYtDSGAp k8S/SWegnB2MEkcWzWKFqXq4aBVUYiOjxK+zzWwgCTYBI4lv72eygNgiAmoSm14tYgexmQWe MUq8310MYgsLREksvnEELM4ioCrRs3ICM4jNK2AvcaX1IBvEAnmJXW0XwZZxCjhIzPy2EMwW Aqo52z+HHWSxhMAyVomzP9YxQTQLSpyc+YQFYpmExMEXL5ghGtQlTi6ZzzSBUWgWkrJZSMoW MDKtYhTLzcwpzk1PLTA00StOzEvJLM7WS87P3cQIiYLCHYyvb3sfYhTgYFTi4T0huzhCiDWx rLgy9xCjBAezkgivhgNQiDclsbIqtSg/vqg0J7X4EKM0B4uSOG+W7NQIIYH0xJLU7NTUgtQi mCwTB6dUA+P8N5OXz/jbXbZMdtJD3qrUY+16Jc05GUdWHT6trHZkxkdJ7pOcmnorrzVJrz16 Yb2wsszMb2p9mar31kq/Pr968qLPS9cZ7F8qtNdtyzG3w+JyvD7TFsuvPcNVvyNpyt9dz1z8 pky4Nt39QNAby7rvKl8XVjWsULn+n5u38ETmuoD9s++sLovrU2Ipzkg01GIuKk4EAE3FWK9+ AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrFJMWRmVeSWpSXmKPExsXCtZGTTfdF0OIIg66f/BYHf7YxWvy/+5zJ 4uX5D6wWK1cfZbJ4dqiXxeL0owPsFv/WdrM4sHvsvtnA5vFs+mEmj0fPzzF6fHx6i8Xj/b6r bB7T1pxn8vi8SS6APYrLJiU1J7MstUjfLoEr4/S9xYwFs5Qrpr3dy9jAeFSmi5GTQ0LAROLh olVsXYxcHEIC6xklfp9/ywiSYBMwkvj2fiYLiC0ioCax6dUidpAiZoEXjBI/Pn0ASwgLREks vnGEHcRmEVCV6Fk5gbmLkYODV8BeYtVZT4gF8hK72i6ygticAg4SM78tBLOFgErO9s9hn8DI vYCRYRWjWG5mTnFuemaBoZFecWJeSmZxtl5yfu4mRkjoRO1gvD7R/BAjEwenVANjsp+c0bY/ 666971o4ubQqvdbxamxo5rmQm9NcC+t7XdzajL+cO+pjv/Hm6+cMx96WMPhvFFqgsH9v9+TP 3mERc/u0fr/XKPAUmDVn9dkpif51E3/NW+z8lPHNL6ffWpUbKlhWb946i+/lNNdjxknbVP9+ bV46K/omH7foeft+7/dRczuFTGv+KrEUZyQaajEXFScCAPyemrDNAQAA MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Bart Van Assche Cc: Steve Wise Cc: Chuck Lever Cc: Christoph Hellwig Cc: Max Gurtovoy Reviewed-by: Steve Wise Reviewed-by: Leon Romanovsky --- drivers/infiniband/core/cq.c | 4 ++-- drivers/infiniband/core/verbs.c | 35 +++++++++++++++-------------------- 2 files changed, 17 insertions(+), 22 deletions(-) diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c index a754fc727de5..48242785b323 100644 --- a/drivers/infiniband/core/cq.c +++ b/drivers/infiniband/core/cq.c @@ -58,8 +58,8 @@ static int __ib_process_cq(struct ib_cq *cq, int budget) * %IB_POLL_DIRECT CQ. It does not offload CQ processing to a different * context and does not ask for completion interrupts from the HCA. * - * Note: for compatibility reasons -1 can be passed in %budget for unlimited - * polling. Do not use this feature in new code, it will be removed soon. + * Note: do not pass -1 as %budget unless it is guaranteed that the number + * of completions that will be processed is small. */ int ib_process_cq_direct(struct ib_cq *cq, int budget) { diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index 71580cc28c9e..42f8927b542c 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -1949,17 +1949,12 @@ static void ib_drain_qp_done(struct ib_cq *cq, struct ib_wc *wc) */ static void __ib_drain_sq(struct ib_qp *qp) { + struct ib_cq *cq = qp->send_cq; struct ib_qp_attr attr = { .qp_state = IB_QPS_ERR }; struct ib_drain_cqe sdrain; struct ib_send_wr swr = {}, *bad_swr; int ret; - if (qp->send_cq->poll_ctx == IB_POLL_DIRECT) { - WARN_ONCE(qp->send_cq->poll_ctx == IB_POLL_DIRECT, - "IB_POLL_DIRECT poll_ctx not supported for drain\n"); - return; - } - swr.wr_cqe = &sdrain.cqe; sdrain.cqe.done = ib_drain_qp_done; init_completion(&sdrain.done); @@ -1976,7 +1971,11 @@ static void __ib_drain_sq(struct ib_qp *qp) return; } - wait_for_completion(&sdrain.done); + if (cq->poll_ctx == IB_POLL_DIRECT) + while (wait_for_completion_timeout(&sdrain.done, HZ / 10) <= 0) + ib_process_cq_direct(cq, -1); + else + wait_for_completion(&sdrain.done); } /* @@ -1984,17 +1983,12 @@ static void __ib_drain_sq(struct ib_qp *qp) */ static void __ib_drain_rq(struct ib_qp *qp) { + struct ib_cq *cq = qp->recv_cq; struct ib_qp_attr attr = { .qp_state = IB_QPS_ERR }; struct ib_drain_cqe rdrain; struct ib_recv_wr rwr = {}, *bad_rwr; int ret; - if (qp->recv_cq->poll_ctx == IB_POLL_DIRECT) { - WARN_ONCE(qp->recv_cq->poll_ctx == IB_POLL_DIRECT, - "IB_POLL_DIRECT poll_ctx not supported for drain\n"); - return; - } - rwr.wr_cqe = &rdrain.cqe; rdrain.cqe.done = ib_drain_qp_done; init_completion(&rdrain.done); @@ -2011,7 +2005,11 @@ static void __ib_drain_rq(struct ib_qp *qp) return; } - wait_for_completion(&rdrain.done); + if (cq->poll_ctx == IB_POLL_DIRECT) + while (wait_for_completion_timeout(&rdrain.done, HZ / 10) <= 0) + ib_process_cq_direct(cq, -1); + else + wait_for_completion(&rdrain.done); } /** @@ -2028,8 +2026,7 @@ static void __ib_drain_rq(struct ib_qp *qp) * ensure there is room in the CQ and SQ for the drain work request and * completion. * - * allocate the CQ using ib_alloc_cq() and the CQ poll context cannot be - * IB_POLL_DIRECT. + * allocate the CQ using ib_alloc_cq(). * * ensure that there are no other contexts that are posting WRs concurrently. * Otherwise the drain is not guaranteed. @@ -2057,8 +2054,7 @@ EXPORT_SYMBOL(ib_drain_sq); * ensure there is room in the CQ and RQ for the drain work request and * completion. * - * allocate the CQ using ib_alloc_cq() and the CQ poll context cannot be - * IB_POLL_DIRECT. + * allocate the CQ using ib_alloc_cq(). * * ensure that there are no other contexts that are posting WRs concurrently. * Otherwise the drain is not guaranteed. @@ -2082,8 +2078,7 @@ EXPORT_SYMBOL(ib_drain_rq); * ensure there is room in the CQ(s), SQ, and RQ for drain work requests * and completions. * - * allocate the CQs using ib_alloc_cq() and the CQ poll context cannot be - * IB_POLL_DIRECT. + * allocate the CQs using ib_alloc_cq(). * * ensure that there are no other contexts that are posting WRs concurrently. * Otherwise the drain is not guaranteed.