From patchwork Fri Feb 8 21:53:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Parvi Kaustubhi (pkaustub)" X-Patchwork-Id: 10803817 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 354AB13BF for ; Fri, 8 Feb 2019 21:54:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 21D1A285FF for ; Fri, 8 Feb 2019 21:54:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1264F2872E; Fri, 8 Feb 2019 21:54:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E988B285FF for ; Fri, 8 Feb 2019 21:54:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726979AbfBHVyG (ORCPT ); Fri, 8 Feb 2019 16:54:06 -0500 Received: from rcdn-iport-3.cisco.com ([173.37.86.74]:48030 "EHLO rcdn-iport-3.cisco.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726791AbfBHVyG (ORCPT ); Fri, 8 Feb 2019 16:54:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=1386; q=dns/txt; s=iport; t=1549662845; x=1550872445; h=from:to:cc:subject:date:message-id; bh=fyFBpbLzdSOroOsKj2BH/j10T/Q5GlcDFv3k0jgzZE0=; b=X/n4SqT6EsSIjFEEfH4i8VftJPYUxHxYr4W6waSEuje0PSmJpZ0Vc9+8 iz34dYWgF551HNN5b/IMibTRO5juiY4b8Oh9TdLEUooc9RN9aYCxZOsxH whRQqkWp/vxepRr9mFbgANI9V/SR6LSyD6siblzAk2L4942putC1kGkyy c=; X-IronPort-AV: E=Sophos;i="5.58,348,1544486400"; d="scan'208";a="504321564" Received: from alln-core-6.cisco.com ([173.36.13.139]) by rcdn-iport-3.cisco.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Feb 2019 21:54:04 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by alln-core-6.cisco.com (8.15.2/8.15.2) with ESMTP id x18Ls4bU015492; Fri, 8 Feb 2019 21:54:04 GMT Received: by cisco.com (Postfix, from userid 478433) id 473CE20F2001; Fri, 8 Feb 2019 13:54:04 -0800 (PST) From: Parvi Kaustubhi To: linux-rdma@vger.kernel.org Cc: gvaradar@cisco.com, benve@cisco.com, tinamdar@cisco.com, neescoba@cisco.com, jsquyres@cisco.com, ravianan@cisco.com, pkaustub@cisco.com Subject: [PATCH for-next v3 1/2] IB/usnic: Fix locking when unregistering Date: Fri, 8 Feb 2019 13:53:43 -0800 Message-Id: <1549662824-23439-1-git-send-email-pkaustub@cisco.com> X-Mailer: git-send-email 2.7.0 X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: alln-core-6.cisco.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Move the call to usnic_ib_device_remove after usnic_ib_ibdev_list_lock has been released. Signed-off-by: Parvi Kaustubhi --- Changelog: v2->v3: * Jason: drivers should not hold any locks while calling ib_unregister_device() * Jason: use https://github.com/jgunthorpe/linux/commits/device_locking_cleanup v1->v2: * Have notifier blocks in usnic_ib_dev instead of using workqueue to defer event handling. --- drivers/infiniband/hw/usnic/usnic_ib_main.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/usnic/usnic_ib_main.c b/drivers/infiniband/hw/usnic/usnic_ib_main.c index 6a69968..9529a08 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib_main.c +++ b/drivers/infiniband/hw/usnic/usnic_ib_main.c @@ -474,15 +474,17 @@ static void usnic_ib_undiscover_pf(struct kref *kref) &usnic_ib_ibdev_list, ib_dev_link) { if (us_ibdev->pdev == dev) { list_del(&us_ibdev->ib_dev_link); - usnic_ib_device_remove(us_ibdev); found = true; break; } } - WARN(!found, "Failed to remove PF %s\n", pci_name(dev)); mutex_unlock(&usnic_ib_ibdev_list_lock); + if (found) + usnic_ib_device_remove(us_ibdev); + else + WARN(1, "Failed to remove PF %s\n", pci_name(dev)); } static struct usnic_ib_dev *usnic_ib_discover_pf(struct usnic_vnic *vnic) From patchwork Fri Feb 8 21:53:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Parvi Kaustubhi (pkaustub)" X-Patchwork-Id: 10803819 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1C2D513BF for ; Fri, 8 Feb 2019 21:54:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0C4C8285FF for ; Fri, 8 Feb 2019 21:54:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F36D92872E; Fri, 8 Feb 2019 21:54:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 97305285FF for ; Fri, 8 Feb 2019 21:54:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726791AbfBHVyM (ORCPT ); Fri, 8 Feb 2019 16:54:12 -0500 Received: from rcdn-iport-9.cisco.com ([173.37.86.80]:64102 "EHLO rcdn-iport-9.cisco.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726704AbfBHVyL (ORCPT ); Fri, 8 Feb 2019 16:54:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=3108; q=dns/txt; s=iport; t=1549662850; x=1550872450; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=oY7qHTeHpMpGsj/+VFenlGWS5O47wyOQf+ddn4lhR68=; b=JRLHRNea3wKtxyTKb46UYANT0+wiDWH7oaGSw9R8NHNePBIvOYVsYgnP Zldx9O7iFi2cmR17vK6cu/RLMx0KPG5oFqDArdLvkXmpzehlv5AF8YyQH yx9pJAFdMUL7xTC20i0kbbE8xUbtfjptpayuLcRuW1uVK9dt2ccGa+aTc A=; X-IronPort-AV: E=Sophos;i="5.58,348,1544486400"; d="scan'208";a="429990590" Received: from alln-core-11.cisco.com ([173.36.13.133]) by rcdn-iport-9.cisco.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Feb 2019 21:54:10 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by alln-core-11.cisco.com (8.15.2/8.15.2) with ESMTP id x18Ls9st007076; Fri, 8 Feb 2019 21:54:09 GMT Received: by cisco.com (Postfix, from userid 478433) id A5C6E20F2001; Fri, 8 Feb 2019 13:54:09 -0800 (PST) From: Parvi Kaustubhi To: linux-rdma@vger.kernel.org Cc: gvaradar@cisco.com, benve@cisco.com, tinamdar@cisco.com, neescoba@cisco.com, jsquyres@cisco.com, ravianan@cisco.com, pkaustub@cisco.com Subject: [PATCH for-next v3 2/2] IB/usnic: fix deadlock Date: Fri, 8 Feb 2019 13:53:44 -0800 Message-Id: <1549662824-23439-2-git-send-email-pkaustub@cisco.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1549662824-23439-1-git-send-email-pkaustub@cisco.com> References: <1549662824-23439-1-git-send-email-pkaustub@cisco.com> X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: alln-core-11.cisco.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP There is a dead lock in usnic ib_register and netdev_notify path. usnic_ib_discover_pf() | mutex_lock(&usnic_ib_ibdev_list_lock); | usnic_ib_device_add(); | ib_register_device() | usnic_ib_query_port() | mutex_lock(&us_ibdev->usdev_lock); | ib_get_eth_speed() | rtnl_lock() order of lock: &usnic_ib_ibdev_list_lock -> usdev_lock -> rtnl_lock rtnl_lock() | usnic_ib_netdevice_event() | mutex_lock(&usnic_ib_ibdev_list_lock); order of lock: rtnl_lock -> &usnic_ib_ibdev_list_lock Solution is to use ib_device_get_by_netdev() to lookup ib_dev while handling netdev/ inet events. Signed-off-by: Parvi Kaustubhi Reviewed-by: Govindarajulu Varadarajan Reviewed-by: Tanmay Inamdar --- Changelog: v2->v3: * Jason: drivers should not hold any locks while calling ib_unregister_device() * Jason: use https://github.com/jgunthorpe/linux/commits/device_locking_cleanup v1->v2: * Have notifier blocks in usnic_ib_dev instead of using workqueue to defer event handling. --- drivers/infiniband/hw/usnic/usnic_ib_main.c | 34 ++++++++++++++--------------- 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/drivers/infiniband/hw/usnic/usnic_ib_main.c b/drivers/infiniband/hw/usnic/usnic_ib_main.c index 9529a08..53d6c12 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib_main.c +++ b/drivers/infiniband/hw/usnic/usnic_ib_main.c @@ -216,18 +216,17 @@ static int usnic_ib_netdevice_event(struct notifier_block *notifier, unsigned long event, void *ptr) { struct usnic_ib_dev *us_ibdev; + struct ib_device *ibdev; struct net_device *netdev = netdev_notifier_info_to_dev(ptr); - mutex_lock(&usnic_ib_ibdev_list_lock); - list_for_each_entry(us_ibdev, &usnic_ib_ibdev_list, ib_dev_link) { - if (us_ibdev->netdev == netdev) { - usnic_ib_handle_usdev_event(us_ibdev, event); - break; - } - } - mutex_unlock(&usnic_ib_ibdev_list_lock); - + ibdev = ib_device_get_by_netdev(netdev, RDMA_DRIVER_USNIC); + if (!ibdev) + goto exit; + else + us_ibdev = container_of(ibdev, struct usnic_ib_dev, ib_dev); + usnic_ib_handle_usdev_event(us_ibdev, event); +exit: return NOTIFY_DONE; } @@ -282,16 +281,15 @@ static int usnic_ib_inetaddr_event(struct notifier_block *notifier, struct usnic_ib_dev *us_ibdev; struct in_ifaddr *ifa = ptr; struct net_device *netdev = ifa->ifa_dev->dev; + struct ib_device *ibdev; - mutex_lock(&usnic_ib_ibdev_list_lock); - list_for_each_entry(us_ibdev, &usnic_ib_ibdev_list, ib_dev_link) { - if (us_ibdev->netdev == netdev) { - usnic_ib_handle_inet_event(us_ibdev, event, ptr); - break; - } - } - mutex_unlock(&usnic_ib_ibdev_list_lock); - + ibdev = ib_device_get_by_netdev(netdev, RDMA_DRIVER_USNIC); + if (!ibdev) + goto exit; + else + us_ibdev = container_of(ibdev, struct usnic_ib_dev, ib_dev); + usnic_ib_handle_inet_event(us_ibdev, event, ptr); +exit: return NOTIFY_DONE; } static struct notifier_block usnic_ib_inetaddr_notifier = {