From patchwork Fri Aug 25 19:05:47 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Boyer X-Patchwork-Id: 9922689 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 92FA26022E for ; Fri, 25 Aug 2017 19:08:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8230828497 for ; Fri, 25 Aug 2017 19:08:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 76D48284AF; Fri, 25 Aug 2017 19:08:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 09C5828497 for ; Fri, 25 Aug 2017 19:08:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755324AbdHYTI1 (ORCPT ); Fri, 25 Aug 2017 15:08:27 -0400 Received: from esa7.dell-outbound.iphmx.com ([68.232.153.96]:45956 "EHLO esa7.dell-outbound.iphmx.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755294AbdHYTI0 (ORCPT ); Fri, 25 Aug 2017 15:08:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=dell.com; i=@dell.com; q=dns/txt; s=smtpout; t=1503687746; x=1535223746; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=0Ae4rDiXcIoICQ6ZLFmyISptLX1hwH7Pcxvsvhz8PGU=; b=ZTxEUTadMSHEoL3JGR/VbnH1sH6nS22UVM0Vf9JTT/IL+O2KLDfIWJMY 2FEijUcc9CcNqrYpOaxiXZWmA8pX5EXgOIzWZoeI1e4LXujApDRsS+DiO h5uJEJP2m96+bpQl7iFn7WhCOD75a6rHagqW9OQYjvXzU/gAFjzCQXkRi M=; Received: from esa2.dell-outbound2.iphmx.com ([68.232.153.202]) by esa7.dell-outbound.iphmx.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 25 Aug 2017 14:01:10 -0500 Received: from mailuogwdur.emc.com ([128.221.224.79]) by esa2.dell-outbound2.iphmx.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 26 Aug 2017 00:56:51 +0600 Received: from maildlpprd51.lss.emc.com (maildlpprd51.lss.emc.com [10.106.48.155]) by mailuogwprd53.lss.emc.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.0) with ESMTP id v7PJ6s8i017211 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 25 Aug 2017 15:06:56 -0400 X-DKIM: OpenDKIM Filter v2.4.3 mailuogwprd53.lss.emc.com v7PJ6s8i017211 Received: from mailapphubprd52.lss.emc.com (emcmail.lss.emc.com [10.106.83.171]) by maildlpprd51.lss.emc.com (RSA Interceptor); Fri, 25 Aug 2017 15:06:12 -0400 Received: from hopcyc-boyera-1.corp.emc.com (hopcyc-boyera-1.cec.lab.emc.com [10.244.91.191]) by mailapphubprd52.lss.emc.com (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.0) with ESMTP id v7PJ6LgB017141; Fri, 25 Aug 2017 15:06:39 -0400 From: Andrew Boyer To: monis@mellanox.com, yonatanc@mellanox.com, linux-rdma@vger.kernel.org Cc: Andrew Boyer Subject: [PATCH v1 02/11] IB/rxe: Disable completion upcalls when a CQ is destroyed Date: Fri, 25 Aug 2017 15:05:47 -0400 Message-Id: <1503687956-7110-3-git-send-email-andrew.boyer@dell.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1503687956-7110-1-git-send-email-andrew.boyer@dell.com> References: <1500989968-30889-1-git-send-email-andrew.boyer@dell.com> <1503687956-7110-1-git-send-email-andrew.boyer@dell.com> X-RSA-Classifications: public X-Sentrion-Hostname: mailuogwprd53.lss.emc.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This prevents the stack from accessing userspace objects while they are being torn down. One possible sequence of events: - Userspace program exits - ib_uverbs_cleanup_ucontext() runs, calling ib_destroy_qp(), ib_destroy_cq(), etc. and releasing/freeing the UCQ - The QP still has tasklets running, so it isn't destroyed yet - The CQ is referenced by the QP, so the CQ isn't destroyed yet - The UCQ is kfree()'d anyway - A send work request completes - rxe_send_complete() calls cq->ibcq.comp_handler() - ib_uverbs_comp_handler() runs and crashes; the event queue is checked for is_closed, but it has no way to check the ib_ucq_object before accessing it The reference counting on the CQ doesn't protect against this since the CQ hasn't been destroyed yet. There's no available interface to deregister the UCQ from the CQ, and it didn't appear that attempting to add reference counting to the UCQ was going to be a good way to go since this solution is much simpler. Fixes: 8700e3e7c485 ("Soft RoCE driver") Signed-off-by: Andrew Boyer --- drivers/infiniband/sw/rxe/rxe_cq.c | 19 +++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_loc.h | 2 ++ drivers/infiniband/sw/rxe/rxe_verbs.c | 2 ++ drivers/infiniband/sw/rxe/rxe_verbs.h | 1 + 4 files changed, 24 insertions(+) diff --git a/drivers/infiniband/sw/rxe/rxe_cq.c b/drivers/infiniband/sw/rxe/rxe_cq.c index 49fe42c..c4aabf7 100644 --- a/drivers/infiniband/sw/rxe/rxe_cq.c +++ b/drivers/infiniband/sw/rxe/rxe_cq.c @@ -69,6 +69,14 @@ int rxe_cq_chk_attr(struct rxe_dev *rxe, struct rxe_cq *cq, static void rxe_send_complete(unsigned long data) { struct rxe_cq *cq = (struct rxe_cq *)data; + unsigned long flags; + + spin_lock_irqsave(&cq->cq_lock, flags); + if (cq->is_dying) { + spin_unlock_irqrestore(&cq->cq_lock, flags); + return; + } + spin_unlock_irqrestore(&cq->cq_lock, flags); cq->ibcq.comp_handler(&cq->ibcq, cq->ibcq.cq_context); } @@ -97,6 +105,8 @@ int rxe_cq_from_init(struct rxe_dev *rxe, struct rxe_cq *cq, int cqe, if (udata) cq->is_user = 1; + cq->is_dying = false; + tasklet_init(&cq->comp_task, rxe_send_complete, (unsigned long)cq); spin_lock_init(&cq->cq_lock); @@ -156,6 +166,15 @@ int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited) return 0; } +void rxe_cq_disable(struct rxe_cq *cq) +{ + unsigned long flags; + + spin_lock_irqsave(&cq->cq_lock, flags); + cq->is_dying = true; + spin_unlock_irqrestore(&cq->cq_lock, flags); +} + void rxe_cq_cleanup(struct rxe_pool_entry *arg) { struct rxe_cq *cq = container_of(arg, typeof(*cq), pelem); diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index d6299ed..64f8fa1 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -64,6 +64,8 @@ int rxe_cq_from_init(struct rxe_dev *rxe, struct rxe_cq *cq, int cqe, int rxe_cq_post(struct rxe_cq *cq, struct rxe_cqe *cqe, int solicited); +void rxe_cq_disable(struct rxe_cq *cq); + void rxe_cq_cleanup(struct rxe_pool_entry *arg); /* rxe_mcast.c */ diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index af90a7d..02a39e8 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -960,6 +960,8 @@ static int rxe_destroy_cq(struct ib_cq *ibcq) { struct rxe_cq *cq = to_rcq(ibcq); + rxe_cq_disable(cq); + rxe_drop_ref(cq); return 0; } diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 5a180fb..b09a9e2 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -89,6 +89,7 @@ struct rxe_cq { struct rxe_queue *queue; spinlock_t cq_lock; u8 notify; + bool is_dying; int is_user; struct tasklet_struct comp_task; };