From patchwork Wed Dec 5 15:14:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Wise X-Patchwork-Id: 10714745 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ECCE817D5 for ; Wed, 5 Dec 2018 18:51:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DF85C2E1CA for ; Wed, 5 Dec 2018 18:51:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DDCDA2E1E6; Wed, 5 Dec 2018 18:51:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00,DATE_IN_PAST_03_06, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C4FB2E1CA for ; Wed, 5 Dec 2018 18:51:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727763AbeLESvm (ORCPT ); Wed, 5 Dec 2018 13:51:42 -0500 Received: from opengridcomputing.com ([72.48.214.68]:51386 "EHLO smtp.opengridcomputing.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727297AbeLESvl (ORCPT ); Wed, 5 Dec 2018 13:51:41 -0500 Received: by smtp.opengridcomputing.com (Postfix, from userid 503) id B88FF227A0; Wed, 5 Dec 2018 12:51:40 -0600 (CST) Message-Id: <6d6a4bb000730db4fcd8e54e9c7a1151c68dba27.1544033819.git.swise@opengridcomputing.com> In-Reply-To: References: From: Steve Wise Date: Wed, 5 Dec 2018 07:14:15 -0800 Subject: [PATCH v6 1/4] rdma_rxe: serialize device removal To: dledford@redhat.com, jgg@mellanox.com Cc: linux-rdma@vger.kernel.org, BMT@zurich.ibm.com, leon@kernel.org, markb@mellanox.com, yanjun.zhu@oracle.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The rxe device can be removed, which includes unregistering with the rdma core, from 3 places: a netdev notifier call, the sysfs handler used to delete a rxe device, and module unload. Currently these are not synchronized to ensure that the device is unregistered once and the memory only freed once. This commit adds proper serialization. Device removal and deregistration is consolidated into rxe_net_remove() which uses dev_list_lock to serialize removal from rxe_dev_list and ensures only 1 deregister. Functions net_to_rxe() and get_rxe_by_name() both now take a kref reference with dev_list_lock held to ensure that during a race between 2 removes, the rxe struct memory remains around until both racers release the reference. So now callers to these functions must call rxe_dev_put() when they are done using the rxe pointer. Signed-off-by: Steve Wise --- drivers/infiniband/sw/rxe/rxe.c | 8 -------- drivers/infiniband/sw/rxe/rxe.h | 1 - drivers/infiniband/sw/rxe/rxe_net.c | 26 ++++++++++++++++++++++---- drivers/infiniband/sw/rxe/rxe_net.h | 1 + drivers/infiniband/sw/rxe/rxe_sysfs.c | 4 ++-- 5 files changed, 25 insertions(+), 15 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 383e65c7bbc0..971f0862cefe 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -331,14 +331,6 @@ int rxe_add(struct rxe_dev *rxe, unsigned int mtu) return err; } -/* called by the ifc layer to remove a device */ -void rxe_remove(struct rxe_dev *rxe) -{ - rxe_unregister_device(rxe); - - rxe_dev_put(rxe); -} - static int __init rxe_module_init(void) { int err; diff --git a/drivers/infiniband/sw/rxe/rxe.h b/drivers/infiniband/sw/rxe/rxe.h index 8f79bd86d033..e0c0dce80bbf 100644 --- a/drivers/infiniband/sw/rxe/rxe.h +++ b/drivers/infiniband/sw/rxe/rxe.h @@ -96,7 +96,6 @@ static inline u32 rxe_crc32(struct rxe_dev *rxe, void rxe_set_mtu(struct rxe_dev *rxe, unsigned int dev_mtu); int rxe_add(struct rxe_dev *rxe, unsigned int mtu); -void rxe_remove(struct rxe_dev *rxe); void rxe_remove_all(void); void rxe_rcv(struct sk_buff *skb); diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index b26a8141f3ed..6dc1a5b20e31 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -57,6 +57,7 @@ struct rxe_dev *net_to_rxe(struct net_device *ndev) list_for_each_entry(rxe, &rxe_dev_list, list) { if (rxe->ndev == ndev) { found = rxe; + kref_get(&rxe->ref_cnt); break; } } @@ -74,6 +75,7 @@ struct rxe_dev *get_rxe_by_name(const char *name) list_for_each_entry(rxe, &rxe_dev_list, list) { if (!strcmp(name, dev_name(&rxe->ib_dev.dev))) { found = rxe; + kref_get(&rxe->ref_cnt); break; } } @@ -573,6 +575,21 @@ struct rxe_dev *rxe_net_add(struct net_device *ndev) return rxe; } +void rxe_net_remove(struct rxe_dev *rxe) +{ + bool already_removed; + + spin_lock_bh(&dev_list_lock); + already_removed = list_empty(&rxe->list); + list_del_init(&rxe->list); + spin_unlock_bh(&dev_list_lock); + + if (!already_removed) { + rxe_unregister_device(rxe); + rxe_dev_put(rxe); + } +} + void rxe_remove_all(void) { spin_lock_bh(&dev_list_lock); @@ -580,9 +597,10 @@ void rxe_remove_all(void) struct rxe_dev *rxe = list_first_entry(&rxe_dev_list, struct rxe_dev, list); - list_del(&rxe->list); + kref_get(&rxe->ref_cnt); spin_unlock_bh(&dev_list_lock); - rxe_remove(rxe); + rxe_net_remove(rxe); + rxe_dev_put(rxe); spin_lock_bh(&dev_list_lock); } spin_unlock_bh(&dev_list_lock); @@ -637,8 +655,7 @@ static int rxe_notify(struct notifier_block *not_blk, switch (event) { case NETDEV_UNREGISTER: - list_del(&rxe->list); - rxe_remove(rxe); + rxe_net_remove(rxe); break; case NETDEV_UP: rxe_port_up(rxe); @@ -666,6 +683,7 @@ static int rxe_notify(struct notifier_block *not_blk, event, ndev->name); break; } + rxe_dev_put(rxe); out: return NOTIFY_OK; } diff --git a/drivers/infiniband/sw/rxe/rxe_net.h b/drivers/infiniband/sw/rxe/rxe_net.h index 106c586dbb26..222234a8d525 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.h +++ b/drivers/infiniband/sw/rxe/rxe_net.h @@ -44,6 +44,7 @@ struct rxe_recv_sockets { }; struct rxe_dev *rxe_net_add(struct net_device *ndev); +void rxe_net_remove(struct rxe_dev *rxe); int rxe_net_init(void); void rxe_net_exit(void); diff --git a/drivers/infiniband/sw/rxe/rxe_sysfs.c b/drivers/infiniband/sw/rxe/rxe_sysfs.c index 73a19f808e1b..fc52eb3d1856 100644 --- a/drivers/infiniband/sw/rxe/rxe_sysfs.c +++ b/drivers/infiniband/sw/rxe/rxe_sysfs.c @@ -137,8 +137,8 @@ static int rxe_param_set_remove(const char *val, const struct kernel_param *kp) return -EINVAL; } - list_del(&rxe->list); - rxe_remove(rxe); + rxe_net_remove(rxe); + rxe_dev_put(rxe); return 0; }