From patchwork Tue Dec 11 22:15:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Parvi Kaustubhi (pkaustub)" X-Patchwork-Id: 10724935 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 80EBF14E2 for ; Tue, 11 Dec 2018 22:25:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 705DF2AA69 for ; Tue, 11 Dec 2018 22:25:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 64A372AAF5; Tue, 11 Dec 2018 22:25:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E46CE2AA69 for ; Tue, 11 Dec 2018 22:25:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726314AbeLKWZR (ORCPT ); Tue, 11 Dec 2018 17:25:17 -0500 Received: from alln-iport-6.cisco.com ([173.37.142.93]:57843 "EHLO alln-iport-6.cisco.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726355AbeLKWZR (ORCPT ); Tue, 11 Dec 2018 17:25:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=5662; q=dns/txt; s=iport; t=1544567116; x=1545776716; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=JtQAsYr6gLdxlTVrL1jUAoziSGZeQE/FfXC9KkFQHEE=; b=O+alGSviwbZXlag3xKW7OC215iJpFO2v4zwi7l14brZO8ndvgxmFF94N Fvppj0D+PlxQJmTzBqx9I9uXLPVcTHGcWcrShgA/e+xwi5B1ULjPkJTWR PJhbmfkDnEptg3Cfc0StcGjJIwN8IuRrmfqbKsXMc0jvqXdNLoXXyuIPK E=; X-IronPort-AV: E=Sophos;i="5.56,343,1539648000"; d="scan'208";a="211088518" Received: from alln-core-2.cisco.com ([173.36.13.135]) by alln-iport-6.cisco.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 11 Dec 2018 22:15:46 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by alln-core-2.cisco.com (8.15.2/8.15.2) with ESMTP id wBBMFkEV018885; Tue, 11 Dec 2018 22:15:46 GMT Received: by cisco.com (Postfix, from userid 478433) id 5E4F420F2001; Tue, 11 Dec 2018 14:15:46 -0800 (PST) From: Parvi Kaustubhi To: linux-rdma@vger.kernel.org Cc: pkaustub@cisco.com, benve@cisco.com, gvaradar@cisco.com, tinamdar@cisco.com, neescoba@cisco.com, ravianan@cisco.com, jsquyres@cisco.com Subject: [PATCH for-next 1/3] IB/usnic: fix deadlock Date: Tue, 11 Dec 2018 14:15:41 -0800 Message-Id: <1544566543-3395-2-git-send-email-pkaustub@cisco.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1544566543-3395-1-git-send-email-pkaustub@cisco.com> References: <1544566543-3395-1-git-send-email-pkaustub@cisco.com> X-Outbound-SMTP-Client: 10.193.184.48, savbu-usnic-a.cisco.com X-Outbound-Node: alln-core-2.cisco.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Govindarajulu Varadarajan There is a dead lock in usnic ib_register() and netdev_notify() path. usnic_ib_discover_pf() | mutex_lock(&usnic_ib_ibdev_list_lock); | usnic_ib_device_add(); | ib_register_device() | usnic_ib_query_port() | mutex_lock(&us_ibdev->usdev_lock); | ib_get_eth_speed() | rtnl_lock() Order of locking: &usnic_ib_ibdev_list_lock -> usdev_lock -> rtnl_lock rtnl_lock() | usnic_ib_netdevice_event() | mutex_lock(&usnic_ib_ibdev_list_lock); Order of locking: rtnl_lock -> &usnic_ib_ibdev_list_lock The solution is to not handle usnic_ib_netdevice_event() with the rtnl lock held. This commit creates a single threaded workqueue to defer the handling of netdev/inet events. Signed-off-by: Govindarajulu Varadarajan Signed-off-by: Parvi Kaustubhi --- drivers/infiniband/hw/usnic/usnic_ib.h | 6 +++ drivers/infiniband/hw/usnic/usnic_ib_main.c | 60 ++++++++++++++++++++++++----- 2 files changed, 56 insertions(+), 10 deletions(-) diff --git a/drivers/infiniband/hw/usnic/usnic_ib.h b/drivers/infiniband/hw/usnic/usnic_ib.h index 525bf27..aff4eb4 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib.h +++ b/drivers/infiniband/hw/usnic/usnic_ib.h @@ -93,6 +93,12 @@ struct usnic_ib_vf { struct list_head link; }; +struct usnic_work { + struct work_struct work; + unsigned long event; + void *ptr; +}; + static inline struct usnic_ib_dev *to_usdev(struct ib_device *ibdev) { diff --git a/drivers/infiniband/hw/usnic/usnic_ib_main.c b/drivers/infiniband/hw/usnic/usnic_ib_main.c index 413fa57..dd28442 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib_main.c +++ b/drivers/infiniband/hw/usnic/usnic_ib_main.c @@ -71,6 +71,7 @@ static const char usnic_version[] = static DEFINE_MUTEX(usnic_ib_ibdev_list_lock); static LIST_HEAD(usnic_ib_ibdev_list); +static struct workqueue_struct *usnic_notify_work; /* Callback dump funcs */ static int usnic_ib_dump_vf_hdr(void *obj, char *buf, int buf_sz) @@ -212,21 +213,35 @@ static void usnic_ib_handle_usdev_event(struct usnic_ib_dev *us_ibdev, mutex_unlock(&us_ibdev->usdev_lock); } -static int usnic_ib_netdevice_event(struct notifier_block *notifier, - unsigned long event, void *ptr) +static void usnic_ib_netdevice_work(struct work_struct *work) { + struct usnic_work *uswork = container_of(work, struct usnic_work, work); + struct net_device *netdev = netdev_notifier_info_to_dev(uswork->ptr); struct usnic_ib_dev *us_ibdev; - struct net_device *netdev = netdev_notifier_info_to_dev(ptr); - mutex_lock(&usnic_ib_ibdev_list_lock); list_for_each_entry(us_ibdev, &usnic_ib_ibdev_list, ib_dev_link) { if (us_ibdev->netdev == netdev) { - usnic_ib_handle_usdev_event(us_ibdev, event); + usnic_ib_handle_usdev_event(us_ibdev, uswork->event); break; } } mutex_unlock(&usnic_ib_ibdev_list_lock); + kfree(uswork); +} + +static int usnic_ib_netdevice_event(struct notifier_block *nblock, + unsigned long event, void *ptr) +{ + struct usnic_work *uswork; + + uswork = kzalloc(sizeof(*uswork), GFP_ATOMIC); + if (uswork) { + uswork->event = event; + uswork->ptr = ptr; + INIT_WORK(&uswork->work, usnic_ib_netdevice_work); + queue_work(usnic_notify_work, &uswork->work); + } return NOTIFY_DONE; } @@ -276,24 +291,42 @@ static int usnic_ib_handle_inet_event(struct usnic_ib_dev *us_ibdev, return NOTIFY_DONE; } -static int usnic_ib_inetaddr_event(struct notifier_block *notifier, - unsigned long event, void *ptr) +static void usnic_ib_inetaddr_work(struct work_struct *work) { + struct usnic_work *uswork = container_of(work, struct usnic_work, work); struct usnic_ib_dev *us_ibdev; - struct in_ifaddr *ifa = ptr; + struct in_ifaddr *ifa = uswork->ptr; struct net_device *netdev = ifa->ifa_dev->dev; mutex_lock(&usnic_ib_ibdev_list_lock); list_for_each_entry(us_ibdev, &usnic_ib_ibdev_list, ib_dev_link) { if (us_ibdev->netdev == netdev) { - usnic_ib_handle_inet_event(us_ibdev, event, ptr); + usnic_ib_handle_inet_event(us_ibdev, uswork->event, + uswork->ptr); break; } } mutex_unlock(&usnic_ib_ibdev_list_lock); + kfree(uswork); +} + +static int usnic_ib_inetaddr_event(struct notifier_block *nblock, + unsigned long event, void *ptr) +{ + struct usnic_work *uswork; + + uswork = kzalloc(sizeof(*uswork), GFP_ATOMIC); + + if (uswork) { + uswork->ptr = ptr; + uswork->event = event; + INIT_WORK(&uswork->work, usnic_ib_inetaddr_work); + queue_work(usnic_notify_work, &uswork->work); + } return NOTIFY_DONE; } + static struct notifier_block usnic_ib_inetaddr_notifier = { .notifier_call = usnic_ib_inetaddr_event }; @@ -653,10 +686,15 @@ static int __init usnic_ib_init(void) return err; } + usnic_notify_work = create_singlethread_workqueue("usnic_notify_work"); + if (!usnic_notify_work) { + usnic_err("Failed to create notify workqueue"); + goto out_umem_fini; + } err = pci_register_driver(&usnic_ib_pci_driver); if (err) { usnic_err("Unable to register with PCI\n"); - goto out_umem_fini; + goto out_notify_work; } err = register_netdevice_notifier(&usnic_ib_netdevice_notifier); @@ -687,6 +725,8 @@ static int __init usnic_ib_init(void) unregister_netdevice_notifier(&usnic_ib_netdevice_notifier); out_pci_unreg: pci_unregister_driver(&usnic_ib_pci_driver); +out_notify_work: + destroy_workqueue(usnic_notify_work); out_umem_fini: usnic_uiom_fini();