From patchwork Wed Nov 23 17:39:22 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Boyer X-Patchwork-Id: 9444007 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id EB0EE60778 for ; Wed, 23 Nov 2016 17:41:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 06F8A26E5D for ; Wed, 23 Nov 2016 17:41:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EFC4227BE5; Wed, 23 Nov 2016 17:41:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FF8126E5D for ; Wed, 23 Nov 2016 17:41:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S936570AbcKWRli (ORCPT ); Wed, 23 Nov 2016 12:41:38 -0500 Received: from esa5.dell-outbound.iphmx.com ([68.232.153.95]:30559 "EHLO esa5.dell-outbound.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S935158AbcKWRlg (ORCPT ); Wed, 23 Nov 2016 12:41:36 -0500 DomainKey-Signature: s=smtpout; d=dell.com; c=simple; q=dns; h=Received:Received:Received:X-DKIM:Received:Received:From: To:Cc:Subject:Date:Message-Id:X-Mailer:In-Reply-To: References:X-RSA-Classifications:X-Sentrion-Hostname; b=W/doR/GhT92v5MSU+lUp12bTWCVOmazg2gkbGMC4QYbOD8Cv5hr+HOrM m2mj2h5Kv0mQbekmSjcdoD+Vp7Ktm42KbiNjrpjnjoVOYnTW3aALykuct g9huDy5I5Diu7UB+vQabAxqNQvr+QJM9JlgqISqk24FHpuhetHAvsb7Nw I=; DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=dell.com; i=@dell.com; q=dns/txt; s=smtpout; t=1479922896; x=1511458896; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=bru8F8pjn0Lxb3nC2oHvlksYbmE86qdJC6RPic68wt0=; b=E6/mKEgn7/qtJWh8vW8L1FjVk+oGE4+NvUSfGCDsCBid7y4AtEc5sD6n /ZmqkssfDAVQS4lCJNZfZBQv7qsseAftoV/+/5G4iKT97aQuQB7NISOWr uLO87owp6W7zH1f4k5OYLlpzvODqBouolQfD6ax3Pr2cDhXm294BwbL8K w=; Received: from esa1.dell-outbound2.iphmx.com ([68.232.153.201]) by esa5.dell-outbound.iphmx.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 23 Nov 2016 11:40:14 -0600 Received: from mailuogwhop.emc.com ([168.159.213.141]) by esa1.dell-outbound2.iphmx.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 23 Nov 2016 23:40:14 +0600 Received: from maildlpprd04.lss.emc.com (maildlpprd04.lss.emc.com [10.253.24.36]) by mailuogwprd01.lss.emc.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.0) with ESMTP id uANHeBZr022021 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 23 Nov 2016 12:40:12 -0500 X-DKIM: OpenDKIM Filter v2.4.3 mailuogwprd01.lss.emc.com uANHeBZr022021 Received: from mailapphubprd02.lss.emc.com (emcmail.lss.emc.com [10.253.24.52]) by maildlpprd04.lss.emc.com (RSA Interceptor); Wed, 23 Nov 2016 12:39:38 -0500 Received: from hopcyc-boyera-1.corp.emc.com (hopcyc-boyera-1.cec.lab.emc.com [10.244.91.191]) by mailapphubprd02.lss.emc.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.0) with ESMTP id uANHdknj016626; Wed, 23 Nov 2016 12:39:48 -0500 From: Andrew Boyer To: monis@mellanox.com, yonatanc@mellanox.com, linux-rdma@vger.kernel.org Cc: Andrew Boyer Subject: [PATCH v2 6/8] IB/rxe: Add support for IB_CQ_REPORT_MISSED_EVENTS Date: Wed, 23 Nov 2016 12:39:22 -0500 Message-Id: <1479922764-13091-7-git-send-email-andrew.boyer@dell.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1479922764-13091-1-git-send-email-andrew.boyer@dell.com> References: <1479922764-13091-1-git-send-email-andrew.boyer@dell.com> X-RSA-Classifications: public X-Sentrion-Hostname: mailuogwprd01.lss.emc.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Peek at the CQ after arming it so that we can return a hint. This avoids missed completions due to a race between posting CQEs and arming the CQ. For example, CM teardown waits on MAD requests to complete with ib_cq_poll_work(). Without this fix, the last completion might be left on the CQ, hanging the kthread doing the teardown. The console backtraces look like this: [ 4199.911284] Call Trace: [ 4199.911401] [] schedule+0x35/0x80 [ 4199.911556] [] schedule_timeout+0x22f/0x2c0 [ 4199.911727] [] ? __schedule+0x368/0xa20 [ 4199.911891] [] wait_for_completion+0xb3/0x130 [ 4199.912067] [] ? wake_up_q+0x70/0x70 [ 4199.912243] [] cm_destroy_id+0x13d/0x450 [ib_cm] [ 4199.912422] [] ? printk+0x57/0x73 [ 4199.912578] [] ib_destroy_cm_id+0x10/0x20 [ib_cm] [ 4199.912759] [] rdma_destroy_id+0xac/0x340 [rdma_cm] [ 4199.912941] [] 0xffffffffc076f2cc Signed-off-by: Andrew Boyer --- drivers/infiniband/sw/rxe/rxe_verbs.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index 19841c8..de39b0a 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -1007,11 +1007,19 @@ static int rxe_peek_cq(struct ib_cq *ibcq, int wc_cnt) static int rxe_req_notify_cq(struct ib_cq *ibcq, enum ib_cq_notify_flags flags) { struct rxe_cq *cq = to_rcq(ibcq); + unsigned long irq_flags; + int ret = 0; + spin_lock_irqsave(&cq->cq_lock, irq_flags); if (cq->notify != IB_CQ_NEXT_COMP) cq->notify = flags & IB_CQ_SOLICITED_MASK; - return 0; + if ((flags & IB_CQ_REPORT_MISSED_EVENTS) && !queue_empty(cq->queue)) + ret = 1; + + spin_unlock_irqrestore(&cq->cq_lock, irq_flags); + + return ret; } static struct ib_mr *rxe_get_dma_mr(struct ib_pd *ibpd, int access)