From patchwork Wed Feb 13 17:23:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 10810453 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5278E6C2 for ; Wed, 13 Feb 2019 17:23:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3450F2E3FB for ; Wed, 13 Feb 2019 17:23:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 285F52E405; Wed, 13 Feb 2019 17:23:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C3AD22E40E for ; Wed, 13 Feb 2019 17:23:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2392927AbfBMRX1 (ORCPT ); Wed, 13 Feb 2019 12:23:27 -0500 Received: from mail.kernel.org ([198.145.29.99]:53596 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2392923AbfBMRX0 (ORCPT ); Wed, 13 Feb 2019 12:23:26 -0500 Received: from localhost (unknown [77.138.135.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 460292175B; Wed, 13 Feb 2019 17:23:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1550078606; bh=1w2j7WvUsJjqQ3B9QVnSGpjqufwIFr3kCzNtxGMdUPg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=M6m6r8HyMEFJ1xREMiUHd35jpYXv471Pn8sawNDczXJZ9TwT8Pg1wNUNPBdoRE6dF dkWQTN47ETa9PY0jnJyV6GqfNZu0oTY4wg4tVbkeQIuMGPA4iNi0KB/iHUTfLJRHqr Q7TNvOH66DrMyNT8yOgNXQ7jxx404d6NXMLMTo7M= From: Leon Romanovsky To: Doug Ledford , Jason Gunthorpe Cc: Leon Romanovsky , RDMA mailing list , Parav Pandit Subject: [PATCH rdma-next 1/8] RDMA/core: Use simpler device_del() instead of device_unregister() Date: Wed, 13 Feb 2019 19:23:03 +0200 Message-Id: <20190213172310.1681-2-leon@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190213172310.1681-1-leon@kernel.org> References: <20190213172310.1681-1-leon@kernel.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Parav Pandit Instead of holding extra reference using get_device() that device_unregister() releases, simplify it as below. device_add() balance with device_del(). device_initialize() balance with put_device(). Signed-off-by: Parav Pandit Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/device.c | 1 + drivers/infiniband/core/sysfs.c | 7 ++----- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index 868d656f3dda..b9baa4a15b42 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -373,6 +373,7 @@ void ib_dealloc_device(struct ib_device *device) WARN_ON(!xa_empty(&device->client_data)); WARN_ON(refcount_read(&device->refcount)); rdma_restrack_clean(device); + /* Balance with device_initialize */ put_device(&device->dev); } EXPORT_SYMBOL(ib_dealloc_device); diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c index c75692802da8..824bd2c283f2 100644 --- a/drivers/infiniband/core/sysfs.c +++ b/drivers/infiniband/core/sysfs.c @@ -1351,15 +1351,12 @@ int ib_device_register_sysfs(struct ib_device *device) void ib_device_unregister_sysfs(struct ib_device *device) { - /* Hold device until ib_dealloc_device() */ - get_device(&device->dev); - free_port_list_attributes(device); if (device->hw_stats) { kfree(device->hw_stats); free_hsag(&device->dev.kobj, device->hw_stats_ag); } - - device_unregister(&device->dev); + /* Balance with device_add */ + device_del(&device->dev); } From patchwork Wed Feb 13 17:23:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 10810467 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F10F01399 for ; Wed, 13 Feb 2019 17:24:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D388D2DD97 for ; Wed, 13 Feb 2019 17:24:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C50EA2E026; Wed, 13 Feb 2019 17:24:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 572882DD97 for ; Wed, 13 Feb 2019 17:24:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731253AbfBMRYB (ORCPT ); Wed, 13 Feb 2019 12:24:01 -0500 Received: from mail.kernel.org ([198.145.29.99]:53956 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726026AbfBMRYB (ORCPT ); Wed, 13 Feb 2019 12:24:01 -0500 Received: from localhost (unknown [77.138.135.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4540B2175B; Wed, 13 Feb 2019 17:23:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1550078640; bh=YwyxlokRwgFhOy/bbrqLaMYOX4mkTIWF11pySR9L48Q=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pwy/G0tEkLPh/iD2fkYuDwh+Q9qCx6u0osfeaCaJftsEMnffQdT8Nx1ar+DxgTEE8 QnqilNFqnr90Zi9eZ7IoaPE5BsUXfqSsL3Nqw26wgQwAcIfXmQgd6tzrG1ajb2uwtw 4WTJsRT5GfNB4JUeh0jidImfPSlFRuxBg+9xB5f8= From: Leon Romanovsky To: Doug Ledford , Jason Gunthorpe Cc: Leon Romanovsky , RDMA mailing list , Parav Pandit Subject: [PATCH rdma-next 2/8] RDMA/core: Introduce and use ib_setup_port_attrs() Date: Wed, 13 Feb 2019 19:23:04 +0200 Message-Id: <20190213172310.1681-3-leon@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190213172310.1681-1-leon@kernel.org> References: <20190213172310.1681-1-leon@kernel.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Parav Pandit Refactor code for device and port sysfs attributes for reuse. while at it, rename counter part free function to ib_free_port_attrs. Also attribute setup sequence is: (a) port specific init. (b) device stats alloc/init. So for cleanup, follow reverse sequence: (a) device stats dealloc (b) port specific cleanup Signed-off-by: Parav Pandit Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/sysfs.c | 64 ++++++++++++++++++--------------- 1 file changed, 35 insertions(+), 29 deletions(-) diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c index 824bd2c283f2..f32de6f77349 100644 --- a/drivers/infiniband/core/sysfs.c +++ b/drivers/infiniband/core/sysfs.c @@ -1279,17 +1279,17 @@ static const struct attribute_group dev_attr_group = { .attrs = ib_dev_attrs, }; -static void free_port_list_attributes(struct ib_device *device) +static void ib_free_port_attrs(struct ib_device *device) { struct kobject *p, *t; list_for_each_entry_safe(p, t, &device->port_list, entry) { struct ib_port *port = container_of(p, struct ib_port, kobj); + list_del(&p->entry); - if (port->hw_stats) { - kfree(port->hw_stats); + if (port->hw_stats_ag) free_hsag(&port->kobj, port->hw_stats_ag); - } + kfree(port->hw_stats); if (port->pma_table) sysfs_remove_group(p, port->pma_table); @@ -1306,24 +1306,14 @@ static void free_port_list_attributes(struct ib_device *device) kobject_put(device->ports_kobj); } -int ib_device_register_sysfs(struct ib_device *device) +static int ib_setup_port_attrs(struct ib_device *device) { - struct device *class_dev = &device->dev; int ret; int i; - device->groups[0] = &dev_attr_group; - class_dev->groups = device->groups; - - ret = device_add(class_dev); - if (ret) - goto err; - - device->ports_kobj = kobject_create_and_add("ports", &class_dev->kobj); - if (!device->ports_kobj) { - ret = -ENOMEM; - goto err_put; - } + device->ports_kobj = kobject_create_and_add("ports", &device->dev.kobj); + if (!device->ports_kobj) + return -ENOMEM; if (rdma_cap_ib_switch(device)) { ret = add_port(device, 0); @@ -1337,26 +1327,42 @@ int ib_device_register_sysfs(struct ib_device *device) } } - if (device->ops.alloc_hw_stats) - setup_hw_stats(device, NULL, 0); - return 0; err_put: - free_port_list_attributes(device); - device_del(class_dev); -err: + ib_free_port_attrs(device); return ret; } -void ib_device_unregister_sysfs(struct ib_device *device) +int ib_device_register_sysfs(struct ib_device *device) { - free_port_list_attributes(device); + int ret; - if (device->hw_stats) { - kfree(device->hw_stats); - free_hsag(&device->dev.kobj, device->hw_stats_ag); + device->groups[0] = &dev_attr_group; + device->dev.groups = device->groups; + + ret = device_add(&device->dev); + if (ret) + return ret; + + ret = ib_setup_port_attrs(device); + if (ret) { + device_del(&device->dev); + return ret; } + if (device->ops.alloc_hw_stats) + setup_hw_stats(device, NULL, 0); + + return 0; +} + +void ib_device_unregister_sysfs(struct ib_device *device) +{ + if (device->hw_stats_ag) + free_hsag(&device->dev.kobj, device->hw_stats_ag); + kfree(device->hw_stats); + + ib_free_port_attrs(device); /* Balance with device_add */ device_del(&device->dev); } From patchwork Wed Feb 13 17:23:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 10810455 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F8101399 for ; Wed, 13 Feb 2019 17:23:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 12D752E3C3 for ; Wed, 13 Feb 2019 17:23:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0723A2E3FB; Wed, 13 Feb 2019 17:23:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 59ADA2E3C3 for ; Wed, 13 Feb 2019 17:23:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389936AbfBMRXc (ORCPT ); Wed, 13 Feb 2019 12:23:32 -0500 Received: from mail.kernel.org ([198.145.29.99]:53640 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731253AbfBMRXc (ORCPT ); Wed, 13 Feb 2019 12:23:32 -0500 Received: from localhost (unknown [77.138.135.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 66EBA2175B; Wed, 13 Feb 2019 17:23:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1550078611; bh=RPs0vH4I+02i8OojyelMWNNH+RFcf91DlRwMcmbLaPM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YjXB3HE9gR4BbflJwef1F2+EkHiuRLkk/2KvbMJ3rRM9NiWHnQtq2nEbAtvKDQFC2 MO6rrTpqkHCzhlQh/ZLWfsHOiYOhrPc4KwxDzXDIgxJWu3wBxgOJtAt+BC6N4k/+oJ jrctH39/Ze2PQFJxliiQsLq6Hp652nUtYPPa1vpo= From: Leon Romanovsky To: Doug Ledford , Jason Gunthorpe Cc: Leon Romanovsky , RDMA mailing list , Parav Pandit Subject: [PATCH rdma-next 3/8] RDMA/core: Introduce ib_core_device to hold device Date: Wed, 13 Feb 2019 19:23:05 +0200 Message-Id: <20190213172310.1681-4-leon@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190213172310.1681-1-leon@kernel.org> References: <20190213172310.1681-1-leon@kernel.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Parav Pandit In order to support sysfs entries in multiple net namespaces for a rdma device, introduce a ib_core_device whose scope is limited to hold core device and per port sysfs related entires. This is preparation patch so that multiple ib_core_devices in each net namespace can be created in subsequent patch who all can share ib_device. (a) Move sysfs specific fields to ib_core_device. (b) Make sysfs and device life cycle related routines to work on ib_core_device. (c) Introduce and use rdma_init_coredev() helper to initialize coredev fields. Signed-off-by: Parav Pandit Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/core_priv.h | 2 ++ drivers/infiniband/core/device.c | 24 ++++++++++++++++--- drivers/infiniband/core/sysfs.c | 36 ++++++++++++++--------------- include/rdma/ib_verbs.h | 25 +++++++++++++++----- 4 files changed, 60 insertions(+), 27 deletions(-) diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h index a1826f4c2e23..eeabe9ca8427 100644 --- a/drivers/infiniband/core/core_priv.h +++ b/drivers/infiniband/core/core_priv.h @@ -54,6 +54,8 @@ struct pkey_index_qp_list { struct list_head qp_list; }; +extern const struct attribute_group ib_dev_attr_group; + int ib_device_register_sysfs(struct ib_device *device); void ib_device_unregister_sysfs(struct ib_device *device); int ib_device_rename(struct ib_device *ibdev, const char *name); diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index b9baa4a15b42..e602c610ab05 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -318,6 +318,25 @@ static struct class ib_class = { .dev_uevent = ib_device_uevent, }; +static void rdma_init_coredev(struct ib_core_device *coredev, + struct ib_device *dev) +{ + /* This BUILD_BUG_ON is intended to catch layout change + * of union of ib_core_device and device. + * dev must be the first element as ib_core and providers + * driver uses it. Adding anything in ib_core_device before + * device will break this assumption. + */ + BUILD_BUG_ON(offsetof(struct ib_device, coredev.dev) != + offsetof(struct ib_device, dev)); + + coredev->dev.class = &ib_class; + coredev->dev.groups = dev->groups; + device_initialize(&coredev->dev); + coredev->owner = dev; + INIT_LIST_HEAD(&coredev->port_list); +} + /** * _ib_alloc_device - allocate an IB device struct * @size:size of structure to allocate @@ -344,8 +363,8 @@ struct ib_device *_ib_alloc_device(size_t size) return NULL; } - device->dev.class = &ib_class; - device_initialize(&device->dev); + device->groups[0] = &ib_dev_attr_group; + rdma_init_coredev(&device->coredev, device); INIT_LIST_HEAD(&device->event_handler_list); spin_lock_init(&device->event_handler_lock); @@ -355,7 +374,6 @@ struct ib_device *_ib_alloc_device(size_t size) */ xa_init_flags(&device->client_data, XA_FLAGS_ALLOC); init_rwsem(&device->client_data_rwsem); - INIT_LIST_HEAD(&device->port_list); init_completion(&device->unreg_completion); return device; diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c index f32de6f77349..292d07a2aed2 100644 --- a/drivers/infiniband/core/sysfs.c +++ b/drivers/infiniband/core/sysfs.c @@ -1015,8 +1015,9 @@ static void setup_hw_stats(struct ib_device *device, struct ib_port *port, return; } -static int add_port(struct ib_device *device, int port_num) +static int add_port(struct ib_core_device *coredev, int port_num) { + struct ib_device *device = rdma_device_to_ibdev(&coredev->dev); struct ib_port *p; struct ib_port_attr attr; int i; @@ -1034,7 +1035,7 @@ static int add_port(struct ib_device *device, int port_num) p->port_num = port_num; ret = kobject_init_and_add(&p->kobj, &port_type, - device->ports_kobj, + coredev->ports_kobj, "%d", port_num); if (ret) { kfree(p); @@ -1125,7 +1126,7 @@ static int add_port(struct ib_device *device, int port_num) if (device->ops.alloc_hw_stats && port_num) setup_hw_stats(device, p, port_num); - list_add_tail(&p->kobj.entry, &device->port_list); + list_add_tail(&p->kobj.entry, &coredev->port_list); kobject_uevent(&p->kobj, KOBJ_ADD); return 0; @@ -1275,15 +1276,15 @@ static struct attribute *ib_dev_attrs[] = { NULL, }; -static const struct attribute_group dev_attr_group = { +const struct attribute_group ib_dev_attr_group = { .attrs = ib_dev_attrs, }; -static void ib_free_port_attrs(struct ib_device *device) +static void ib_free_port_attrs(struct ib_core_device *coredev) { struct kobject *p, *t; - list_for_each_entry_safe(p, t, &device->port_list, entry) { + list_for_each_entry_safe(p, t, &coredev->port_list, entry) { struct ib_port *port = container_of(p, struct ib_port, kobj); list_del(&p->entry); @@ -1303,25 +1304,27 @@ static void ib_free_port_attrs(struct ib_device *device) kobject_put(p); } - kobject_put(device->ports_kobj); + kobject_put(coredev->ports_kobj); } -static int ib_setup_port_attrs(struct ib_device *device) +static int ib_setup_port_attrs(struct ib_core_device *coredev) { + struct ib_device *device = rdma_device_to_ibdev(&coredev->dev); int ret; int i; - device->ports_kobj = kobject_create_and_add("ports", &device->dev.kobj); - if (!device->ports_kobj) + coredev->ports_kobj = kobject_create_and_add("ports", + &coredev->dev.kobj); + if (!coredev->ports_kobj) return -ENOMEM; if (rdma_cap_ib_switch(device)) { - ret = add_port(device, 0); + ret = add_port(coredev, 0); if (ret) goto err_put; } else { for (i = 1; i <= device->phys_port_cnt; ++i) { - ret = add_port(device, i); + ret = add_port(coredev, i); if (ret) goto err_put; } @@ -1330,7 +1333,7 @@ static int ib_setup_port_attrs(struct ib_device *device) return 0; err_put: - ib_free_port_attrs(device); + ib_free_port_attrs(coredev); return ret; } @@ -1338,14 +1341,11 @@ int ib_device_register_sysfs(struct ib_device *device) { int ret; - device->groups[0] = &dev_attr_group; - device->dev.groups = device->groups; - ret = device_add(&device->dev); if (ret) return ret; - ret = ib_setup_port_attrs(device); + ret = ib_setup_port_attrs(&device->coredev); if (ret) { device_del(&device->dev); return ret; @@ -1362,7 +1362,7 @@ void ib_device_unregister_sysfs(struct ib_device *device) free_hsag(&device->dev.kobj, device->hw_stats_ag); kfree(device->hw_stats); - ib_free_port_attrs(device); + ib_free_port_attrs(&device->coredev); /* Balance with device_add */ device_del(&device->dev); } diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index f042c2d00d1a..30314376d032 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -2534,8 +2534,17 @@ struct ib_device_ops { DECLARE_RDMA_OBJ_SIZE(ib_ucontext); }; -struct rdma_restrack_root; +struct ib_core_device { + /* device must be the first element in structure until, + * union of ib_core_device and device exists in ib_device. + */ + struct device dev; + struct kobject *ports_kobj; + struct list_head port_list; + struct ib_device *owner; /* reach back to owner ib_device */ +}; +struct rdma_restrack_root; struct ib_device { /* Do not access @dma_device directly from ULP nor from HW drivers. */ struct device *dma_device; @@ -2561,16 +2570,17 @@ struct ib_device { struct iw_cm_verbs *iwcm; struct module *owner; - struct device dev; + union { + struct device dev; + struct ib_core_device coredev; + }; + /* First group for device attributes, * Second group for driver provided attributes (optional). * It is NULL terminated array. */ const struct attribute_group *groups[3]; - struct kobject *ports_kobj; - struct list_head port_list; - int uverbs_abi_ver; u64 uverbs_cmd_mask; u64 uverbs_ex_cmd_mask; @@ -4294,7 +4304,10 @@ rdma_set_device_sysfs_group(struct ib_device *dev, */ static inline struct ib_device *rdma_device_to_ibdev(struct device *device) { - return container_of(device, struct ib_device, dev); + struct ib_core_device *coredev = + container_of(device, struct ib_core_device, dev); + + return coredev->owner; } /** From patchwork Wed Feb 13 17:23:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 10810457 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8598D1399 for ; Wed, 13 Feb 2019 17:23:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 681A02E405 for ; Wed, 13 Feb 2019 17:23:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5C1AF2E40C; Wed, 13 Feb 2019 17:23:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 05AE92E40E for ; Wed, 13 Feb 2019 17:23:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389931AbfBMRXh (ORCPT ); Wed, 13 Feb 2019 12:23:37 -0500 Received: from mail.kernel.org ([198.145.29.99]:53676 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389447AbfBMRXh (ORCPT ); Wed, 13 Feb 2019 12:23:37 -0500 Received: from localhost (unknown [77.138.135.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 83FE42175B; Wed, 13 Feb 2019 17:23:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1550078616; bh=b/4PlPXnsuEDED/NHcOisnIF477ZXE3sOe8KTqrgNz0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VDx+AyZdTZz/PLoH3Xsu0mCw3xhzh58khHvkvvA6gu5DIxPqbtFMJCRHKDl8Q9xHL Lgh4wUGP7BJdPjW4pQ9+nlpUrJs0o9/8+aHLLnFWKFCepnBRbyWvZahxMAsfQ4JRAC wAjtrITdzOJoCcS6xO2va/RU/D5vrgawwUjHrUUU= From: Leon Romanovsky To: Doug Ledford , Jason Gunthorpe Cc: Leon Romanovsky , RDMA mailing list , Parav Pandit Subject: [PATCH rdma-next 4/8] RDMA/core: Move device addition deletion to device.c Date: Wed, 13 Feb 2019 19:23:06 +0200 Message-Id: <20190213172310.1681-5-leon@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190213172310.1681-1-leon@kernel.org> References: <20190213172310.1681-1-leon@kernel.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Parav Pandit Move core device addition and removal from sysfs.c to device.c as device.c is more appropriate place for device management. Signed-off-by: Parav Pandit Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/device.c | 9 ++++++++- drivers/infiniband/core/sysfs.c | 9 +-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index e602c610ab05..1bad96c1c139 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -788,11 +788,15 @@ int ib_register_device(struct ib_device *device, const char *name) ib_device_register_rdmacg(device); + ret = device_add(&device->dev); + if (ret) + goto cg_cleanup; + ret = ib_device_register_sysfs(device); if (ret) { dev_warn(&device->dev, "Couldn't register device with driver model\n"); - goto cg_cleanup; + goto dev_cleanup; } ret = enable_device(device); @@ -803,6 +807,8 @@ int ib_register_device(struct ib_device *device, const char *name) sysfs_cleanup: ib_device_unregister_sysfs(device); +dev_cleanup: + device_del(&device->dev); cg_cleanup: ib_device_unregister_rdmacg(device); ib_cache_cleanup_one(device); @@ -822,6 +828,7 @@ void ib_unregister_device(struct ib_device *device) { disable_device(device); ib_device_unregister_sysfs(device); + device_del(&device->dev); ib_device_unregister_rdmacg(device); ib_cache_cleanup_one(device); release_name(device); diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c index 292d07a2aed2..55d7d975393a 100644 --- a/drivers/infiniband/core/sysfs.c +++ b/drivers/infiniband/core/sysfs.c @@ -1341,15 +1341,10 @@ int ib_device_register_sysfs(struct ib_device *device) { int ret; - ret = device_add(&device->dev); + ret = ib_setup_port_attrs(&device->coredev); if (ret) return ret; - ret = ib_setup_port_attrs(&device->coredev); - if (ret) { - device_del(&device->dev); - return ret; - } if (device->ops.alloc_hw_stats) setup_hw_stats(device, NULL, 0); @@ -1363,6 +1358,4 @@ void ib_device_unregister_sysfs(struct ib_device *device) kfree(device->hw_stats); ib_free_port_attrs(&device->coredev); - /* Balance with device_add */ - device_del(&device->dev); } From patchwork Wed Feb 13 17:23:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 10810459 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 487026C2 for ; Wed, 13 Feb 2019 17:23:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2BACF2E3C8 for ; Wed, 13 Feb 2019 17:23:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2007C2E3CA; Wed, 13 Feb 2019 17:23:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AE9DF2E3F2 for ; Wed, 13 Feb 2019 17:23:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2392925AbfBMRXm (ORCPT ); Wed, 13 Feb 2019 12:23:42 -0500 Received: from mail.kernel.org ([198.145.29.99]:53728 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731253AbfBMRXl (ORCPT ); Wed, 13 Feb 2019 12:23:41 -0500 Received: from localhost (unknown [77.138.135.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 59AB02175B; Wed, 13 Feb 2019 17:23:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1550078621; bh=oWyiKOgpAPR0jSYAmt8eRpR/N0W650f+6pDZ+MQr6uc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=2ANN9UcFDXzr33xU22EZf6/JvjlyVhdkJpX6ODfvp8RAMV6qck8+DTt0VSKQxNOvB dt+Y8A4Y5YWy+60A7npMxIWzJpxeRCDlVq/+ypRN99DJV765v0Bx7h/2W+k5WjgIt7 oZZ4t/UYeyGWM+bNlT/JdqtrZ/pMernlsVh9ybOk= From: Leon Romanovsky To: Doug Ledford , Jason Gunthorpe Cc: Leon Romanovsky , RDMA mailing list , Parav Pandit Subject: [PATCH rdma-next 5/8] RDMA/core: Restrict sysfs entries view to init_net Date: Wed, 13 Feb 2019 19:23:07 +0200 Message-Id: <20190213172310.1681-6-leon@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190213172310.1681-1-leon@kernel.org> References: <20190213172310.1681-1-leon@kernel.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Parav Pandit This is a preparation patch to provide isolation of rdma device in a network namespace. As first step, make rdma device visible only in init net namespace. Subsequent patch will enable rdma device visibility back in multiple net namespaces using compat ib_core_device device/sysfs tree. Given that ib subsystem depends on net stack, it needs to be initialized after netdev stack and since it support devices, it needs to be initialized before device subsystem; therefore, change initcall sequence to fs_initcall, so that when ib_core is compiled in kernel image, right init sequence is followed. Signed-off-by: Parav Pandit Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/device.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index 1bad96c1c139..35b4bfec91c3 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -312,10 +312,17 @@ static int ib_device_uevent(struct device *device, return 0; } +static const void *net_namespace(struct device *d) +{ + return &init_net; +} + static struct class ib_class = { .name = "infiniband", .dev_release = ib_device_release, .dev_uevent = ib_device_uevent, + .ns_type = &net_ns_type_operations, + .namespace = net_namespace, }; static void rdma_init_coredev(struct ib_core_device *coredev, @@ -1570,5 +1577,8 @@ static void __exit ib_core_cleanup(void) MODULE_ALIAS_RDMA_NETLINK(RDMA_NL_LS, 4); -subsys_initcall(ib_core_init); +/* ib core relies on netdev stack to first register net_ns_type_operations + * ns kobject type before ib_core initialization. + */ +fs_initcall(ib_core_init); module_exit(ib_core_cleanup); From patchwork Wed Feb 13 17:23:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 10810461 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 308451399 for ; Wed, 13 Feb 2019 17:23:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0F47E2E437 for ; Wed, 13 Feb 2019 17:23:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 01A0E2E3F2; Wed, 13 Feb 2019 17:23:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D3A0D2E410 for ; Wed, 13 Feb 2019 17:23:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389972AbfBMRXr (ORCPT ); Wed, 13 Feb 2019 12:23:47 -0500 Received: from mail.kernel.org ([198.145.29.99]:53766 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2392944AbfBMRXr (ORCPT ); Wed, 13 Feb 2019 12:23:47 -0500 Received: from localhost (unknown [77.138.135.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id F2A612175B; Wed, 13 Feb 2019 17:23:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1550078625; bh=Qsm57MtA4a+4+WAS5e5PEoDs6R9FWPHdqI/9kijqSew=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ItLf53QgI/R7lVoNkndABdAcPyAYr8Wy1whug/bIhX+WG0DqS/nxb09k04ZSLMtOr SXk+WtQkS4D1QKMb/orTc8zrUPgVkk7ayHuC4ZehGdPv4Ww7r66e6KpNFGMeLFHo9H dylGedqe2Yoc54gbHAGY3EVELi0yL2jkAApv9Mgo= From: Leon Romanovsky To: Doug Ledford , Jason Gunthorpe Cc: Leon Romanovsky , RDMA mailing list , Parav Pandit Subject: [PATCH rdma-next 6/8] RDMA/core: Implement compat device/sysfs tree in net namespace Date: Wed, 13 Feb 2019 19:23:08 +0200 Message-Id: <20190213172310.1681-7-leon@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190213172310.1681-1-leon@kernel.org> References: <20190213172310.1681-1-leon@kernel.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Parav Pandit Implement compatibility layer sysfs entries of ib_core so that non init_net net namespaces can also discover rdma devices. Each non init_net net namespace has ib_core_device created in it. Such ib_core_device sysfs tree resembles rdma devices found in init_net namespace. This allows discovering rdma devices in multiple non init_net net namespaces via sysfs entries and helpful to rdma-core userspace. Signed-off-by: Parav Pandit Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/core_priv.h | 3 + drivers/infiniband/core/device.c | 289 +++++++++++++++++++++++++++- include/rdma/ib_verbs.h | 6 + 3 files changed, 294 insertions(+), 4 deletions(-) diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h index eeabe9ca8427..7705aa6861b5 100644 --- a/drivers/infiniband/core/core_priv.h +++ b/drivers/infiniband/core/core_priv.h @@ -333,4 +333,7 @@ int roce_resolve_route_from_path(struct sa_path_rec *rec, const struct ib_gid_attr *attr); struct net_device *rdma_read_gid_attr_ndev_rcu(const struct ib_gid_attr *attr); + +void rdma_init_coredev(struct ib_core_device *coredev, struct ib_device *dev, + struct net *net); #endif /* _CORE_PRIV_H */ diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index 35b4bfec91c3..0ab28ab801af 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -38,6 +38,8 @@ #include #include #include +#include +#include #include #include #include @@ -100,6 +102,50 @@ static DECLARE_RWSEM(clients_rwsem); * be registered. */ #define CLIENT_DATA_REGISTERED XA_MARK_1 + +/** + * ib_compat_device - rdma compat device per net namespace + * @coredev: IB core device + * @id: xarray id to identify the compat device; same id as that of + * net namespace xarray. + */ +struct ib_compat_device { + struct ib_core_device coredev; + u32 id; /* xarray id same as that of rdma net namespace */ +}; + +/** + * rdma_dev_net - rdma net namespace metadata for a net + * @net: Pointer to owner net namespace + * @id: xarray id to identify the net namespace. + */ +struct rdma_dev_net { + possible_net_t net; + u32 id; +}; + +/* + * If netns is registered then the corresponding compat device must also + * be registered. + */ +#define NET_NS_REGISTERED XA_MARK_1 + +static unsigned int rdma_dev_net_id; +/* + * Shadow net namespace entries maintained in xarray, which are referred + * by net life cycle routines (init_net/exit_net) and device life cycle + * routines (reg_dev/unreg_dev). + * Without this shadow list, if device life cycle routines access the + * net stack's net ns list, it can miss out to consider a net ns whose + * init_net() is executed but entry is not part of the net ns list in + * setup_net(). + */ +static DEFINE_XARRAY_FLAGS(rdma_nets, XA_FLAGS_TRACK_FREE); +/* + * rwsem to protect accessing the rdma_nets xarray entries. + */ +static DECLARE_RWSEM(rdma_nets_rwsem); + /* * xarray has this behavior where it won't iterate over NULL values stored in * allocated arrays. So we need our own iterator to see all values stored in @@ -226,6 +272,26 @@ static struct ib_device *__ib_device_get_by_name(const char *name) return NULL; } +static int rename_compat_devs(struct ib_device *device) +{ + struct ib_compat_device *cdev; + unsigned long index; + int ret = 0; + + mutex_lock(&device->compat_devs_mutex); + xa_for_each (&device->compat_devs, index, cdev) { + ret = device_rename(&cdev->coredev.dev, dev_name(&device->dev)); + if (ret) { + dev_warn(&cdev->coredev.dev, + "Fail to rename compatdev to new name %s\n", + dev_name(&device->dev)); + break; + } + } + mutex_unlock(&device->compat_devs_mutex); + return ret; +} + int ib_device_rename(struct ib_device *ibdev, const char *name) { int ret; @@ -245,6 +311,7 @@ int ib_device_rename(struct ib_device *ibdev, const char *name) if (ret) goto out; strlcpy(ibdev->name, name, IB_DEVICE_NAME_MAX); + ret = rename_compat_devs(ibdev); out: up_write(&devices_rwsem); return ret; @@ -314,7 +381,10 @@ static int ib_device_uevent(struct device *device, static const void *net_namespace(struct device *d) { - return &init_net; + struct ib_core_device *coredev = + container_of(d, struct ib_core_device, dev); + + return read_pnet(&coredev->rdma_net); } static struct class ib_class = { @@ -325,8 +395,8 @@ static struct class ib_class = { .namespace = net_namespace, }; -static void rdma_init_coredev(struct ib_core_device *coredev, - struct ib_device *dev) +void rdma_init_coredev(struct ib_core_device *coredev, struct ib_device *dev, + struct net *net) { /* This BUILD_BUG_ON is intended to catch layout change * of union of ib_core_device and device. @@ -342,6 +412,7 @@ static void rdma_init_coredev(struct ib_core_device *coredev, device_initialize(&coredev->dev); coredev->owner = dev; INIT_LIST_HEAD(&coredev->port_list); + write_pnet(&coredev->rdma_net, net); } /** @@ -371,7 +442,7 @@ struct ib_device *_ib_alloc_device(size_t size) } device->groups[0] = &ib_dev_attr_group; - rdma_init_coredev(&device->coredev, device); + rdma_init_coredev(&device->coredev, device, &init_net); INIT_LIST_HEAD(&device->event_handler_list); spin_lock_init(&device->event_handler_lock); @@ -381,6 +452,8 @@ struct ib_device *_ib_alloc_device(size_t size) */ xa_init_flags(&device->client_data, XA_FLAGS_ALLOC); init_rwsem(&device->client_data_rwsem); + xa_init_flags(&device->compat_devs, XA_FLAGS_TRACK_FREE); + mutex_init(&device->compat_devs_mutex); init_completion(&device->unreg_completion); return device; @@ -395,6 +468,7 @@ EXPORT_SYMBOL(_ib_alloc_device); */ void ib_dealloc_device(struct ib_device *device) { + WARN_ON(!xa_empty(&device->compat_devs)); WARN_ON(!xa_empty(&device->client_data)); WARN_ON(refcount_read(&device->refcount)); rdma_restrack_clean(device); @@ -593,6 +667,182 @@ static int ib_security_change(struct notifier_block *nb, unsigned long event, return NOTIFY_OK; } +static void compatdev_release(struct device *dev) +{ + struct ib_compat_device *cdev = + container_of(dev, struct ib_compat_device, coredev.dev); + + kfree(cdev); +} + +static int add_one_compat_dev(struct ib_device *device, + struct rdma_dev_net *rnet) +{ + struct ib_compat_device *cdev; + int ret; + + /* create and add compat device in all namespaces other than + * where it is currently bound to. + */ + if (net_eq(read_pnet(&rnet->net), + read_pnet(&device->coredev.rdma_net))) + return 0; + + /* Whichever path among init_net() or ib_register_device() takes the + * compat_devs_mutex first, will be adding the compat devices. So if its + * already added, don't add again. + */ + mutex_lock(&device->compat_devs_mutex); + cdev = xa_load(&device->compat_devs, rnet->id); + if (cdev) { + /* entry already exist for a net, no need to add again. */ + ret = 0; + goto done; + } + + cdev = kzalloc(sizeof(*cdev), GFP_KERNEL); + if (!cdev) { + ret = -ENOMEM; + goto done; + } + cdev->coredev.dev.parent = device->dev.parent; + cdev->id = rnet->id; + + rdma_init_coredev(&cdev->coredev, device, read_pnet(&rnet->net)); + cdev->coredev.dev.release = compatdev_release; + dev_set_name(&cdev->coredev.dev, "%s", dev_name(&device->dev)); + + ret = device_add(&cdev->coredev.dev); + if (ret) + goto add_err; + + ret = xa_insert(&device->compat_devs, rnet->id, cdev, GFP_KERNEL); + if (ret) + goto insert_err; + + mutex_unlock(&device->compat_devs_mutex); + return 0; + +insert_err: + device_del(&cdev->coredev.dev); +add_err: + put_device(&cdev->coredev.dev); +done: + mutex_unlock(&device->compat_devs_mutex); + return ret; +} + +static void remove_one_compat_dev(struct ib_device *device, u32 id) +{ + struct ib_compat_device *cdev; + + mutex_lock(&device->compat_devs_mutex); + cdev = xa_erase(&device->compat_devs, id); + mutex_unlock(&device->compat_devs_mutex); + if (cdev) { + device_del(&cdev->coredev.dev); + put_device(&cdev->coredev.dev); + } +} + +static void remove_compat_devs(struct ib_device *device) +{ + struct ib_compat_device *cdev; + unsigned long index; + + xa_for_each (&device->compat_devs, index, cdev) + remove_one_compat_dev(device, index); +} + +static int add_compat_devs(struct ib_device *device) +{ + struct rdma_dev_net *rnet = NULL; + unsigned long index; + int ret = 0; + + down_read(&rdma_nets_rwsem); + xa_for_each_marked (&rdma_nets, index, rnet, NET_NS_REGISTERED) { + ret = add_one_compat_dev(device, rnet); + if (ret) + break; + } + up_read(&rdma_nets_rwsem); + return ret; +} + +static void rdma_dev_exit_net(struct net *net) +{ + struct rdma_dev_net *rnet = net_generic(net, rdma_dev_net_id); + struct ib_device *dev; + unsigned long index; + + /* Clear the net for REGISTERED state, so that any new device + * registration doesn't use this net to add compat devices. + */ + down_write(&rdma_nets_rwsem); + xa_clear_mark(&rdma_nets, rnet->id, NET_NS_REGISTERED); + up_write(&rdma_nets_rwsem); + + down_read(&devices_rwsem); + xa_for_each (&devices, index, dev) { + /* hold device reference to not free the device + * while working on removing its compat devices. + */ + get_device(&dev->dev); + /* Release the devices_rwsem so that pontentially + * blocking device_del, doesn't hold the devices_rwsem + * for too long. + */ + up_read(&devices_rwsem); + remove_one_compat_dev(dev, rnet->id); + put_device(&dev->dev); + down_read(&devices_rwsem); + } + up_read(&devices_rwsem); + + down_write(&rdma_nets_rwsem); + xa_erase(&rdma_nets, rnet->id); + up_write(&rdma_nets_rwsem); +} + +static __net_init int rdma_dev_init_net(struct net *net) +{ + struct rdma_dev_net *rnet = net_generic(net, rdma_dev_net_id); + unsigned long index; + struct ib_device *dev; + int ret; + + /* No need to create any compat devices in default init_net. */ + if (net_eq(net, &init_net)) + return 0; + + write_pnet(&rnet->net, net); + + down_write(&rdma_nets_rwsem); + ret = xa_alloc(&rdma_nets, &rnet->id, U32_MAX, rnet, GFP_KERNEL); + if (!ret) + xa_set_mark(&rdma_nets, rnet->id, NET_NS_REGISTERED); + up_write(&rdma_nets_rwsem); + if (ret) + return ret; + + /* Hold devices_rwsem to synchronize with disable_device(), + * rename_device() so that we don't add a compat devices for a device + * which may be undergoing unregistration sequence. + */ + down_read(&devices_rwsem); + xa_for_each_marked (&devices, index, dev, DEVICE_REGISTERED) { + ret = add_one_compat_dev(dev, rnet); + if (ret) + break; + } + up_read(&devices_rwsem); + + if (ret) + rdma_dev_exit_net(net); + return ret; +} + /* * Assign the unique string device name and the unique device index. */ @@ -810,8 +1060,18 @@ int ib_register_device(struct ib_device *device, const char *name) if (ret) goto sysfs_cleanup; + ret = add_compat_devs(device); + if (ret) + goto disable_cleanup; + return 0; +disable_cleanup: + disable_device(device); + /* Cleanup any compat devices which are added by init_net() before + * disabling the device. + */ + remove_compat_devs(device); sysfs_cleanup: ib_device_unregister_sysfs(device); dev_cleanup: @@ -834,6 +1094,11 @@ EXPORT_SYMBOL(ib_register_device); void ib_unregister_device(struct ib_device *device) { disable_device(device); + /* compat devices must be removed after device refcount drops to zero. + * Otherwise init_net() may add more compatdevs after removing + * compat devices and before device is disabled. + */ + remove_compat_devs(device); ib_device_unregister_sysfs(device); device_del(&device->dev); ib_device_unregister_rdmacg(device); @@ -842,6 +1107,13 @@ void ib_unregister_device(struct ib_device *device) } EXPORT_SYMBOL(ib_unregister_device); +static struct pernet_operations rdma_dev_net_ops = { + .init = rdma_dev_init_net, + .exit = rdma_dev_exit_net, + .id = &rdma_dev_net_id, + .size = sizeof(struct rdma_dev_net), +}; + static int assign_client_id(struct ib_client *client) { int ret; @@ -1531,12 +1803,20 @@ static int __init ib_core_init(void) goto err_sa; } + ret = register_pernet_device(&rdma_dev_net_ops); + if (ret) { + pr_warn("Couldn't init compat dev. ret %d\n", ret); + goto err_compat; + } + nldev_init(); rdma_nl_register(RDMA_NL_LS, ibnl_ls_cb_table); roce_gid_mgmt_init(); return 0; +err_compat: + unregister_lsm_notifier(&ibdev_lsm_nb); err_sa: ib_sa_cleanup(); err_mad: @@ -1561,6 +1841,7 @@ static void __exit ib_core_cleanup(void) roce_gid_mgmt_cleanup(); nldev_exit(); rdma_nl_unregister(RDMA_NL_LS); + unregister_pernet_device(&rdma_dev_net_ops); unregister_lsm_notifier(&ibdev_lsm_nb); ib_sa_cleanup(); ib_mad_cleanup(); diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 30314376d032..3e78551b0bfa 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -2539,6 +2539,7 @@ struct ib_core_device { * union of ib_core_device and device exists in ib_device. */ struct device dev; + possible_net_t rdma_net; struct kobject *ports_kobj; struct list_head port_list; struct ib_device *owner; /* reach back to owner ib_device */ @@ -2613,6 +2614,11 @@ struct ib_device { */ refcount_t refcount; struct completion unreg_completion; + + /* Protects compat_devs xarray modifications */ + struct mutex compat_devs_mutex; + /* Maintains compat devices for each net namespace */ + struct xarray compat_devs; }; struct ib_client { From patchwork Wed Feb 13 17:23:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 10810463 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A4D841399 for ; Wed, 13 Feb 2019 17:23:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 899C92DD96 for ; Wed, 13 Feb 2019 17:23:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7E41A2DD8A; Wed, 13 Feb 2019 17:23:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D6AF72DD12 for ; Wed, 13 Feb 2019 17:23:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389979AbfBMRXw (ORCPT ); Wed, 13 Feb 2019 12:23:52 -0500 Received: from mail.kernel.org ([198.145.29.99]:53858 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731253AbfBMRXv (ORCPT ); Wed, 13 Feb 2019 12:23:51 -0500 Received: from localhost (unknown [77.138.135.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1225E21872; Wed, 13 Feb 2019 17:23:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1550078631; bh=rr+M/JB6fYRuugp2IX6QVbmveWoO7iiixeYmblg9sTE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bFdZ9cB/y+rzzgpQBs7UJAQZzylw6ow3zUjFYPFA6RRUR3O+yM5XdLpIwAVU02vCA 1AYzV3gm3HAn979IO6nFKAgKwB4i6/JElgi6w2zw0JJmifXycvyv5Wkq/df+98p/6K 0V4vAV6FBPYY8sRF+Qf3q+Dpn1alCd2YW93Rv6Ys= From: Leon Romanovsky To: Doug Ledford , Jason Gunthorpe Cc: Leon Romanovsky , RDMA mailing list , Parav Pandit Subject: [PATCH rdma-next 7/8] RDMA/core: Add Documentation for ib_core_device Date: Wed, 13 Feb 2019 19:23:09 +0200 Message-Id: <20190213172310.1681-8-leon@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190213172310.1681-1-leon@kernel.org> References: <20190213172310.1681-1-leon@kernel.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Parav Pandit Describe ib_core_device, ib_device association and their existence in net namespaces for backward compatibility, and locking scheme. Signed-off-by: Parav Pandit Signed-off-by: Leon Romanovsky --- Documentation/infiniband/core_devices.txt | 146 ++++++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 Documentation/infiniband/core_devices.txt diff --git a/Documentation/infiniband/core_devices.txt b/Documentation/infiniband/core_devices.txt new file mode 100644 index 000000000000..34f7d5cea54f --- /dev/null +++ b/Documentation/infiniband/core_devices.txt @@ -0,0 +1,146 @@ +Linux RDMA devices and their sysfs entries +------------------------------------------ + +1. Background +-------------- +RDMA networking devices have at least 3 link or transport layers. +(a) InfiniBand +(b) RoCE +(c) iWarp + +These networking devices provide kernel bypass for sending/receiving +data to/from the network. + +There are various modes in which these devices are used along with +other protocols for connection establishment and/or for data transfer. +Such as, +(a) rdmacm for connection establishement and verbs for data transfer. +(b) tcp/ip for connection establishment and verbs for data transfer. + +Additionally rdma devices can be shared among multiple net namespaces. + +It is also desired to have per net namespace rdma devices as the +stack matures. + +sysfs entries are heavily used for device discovery, statistics and network +addresses in rdma stack. + +Therefore, to have minimal impact on backward compatibility for these 3 +transports and to provide forward looking method, the following sysfs +isolation approach is taken. + +2. Design +---------- + +For every rdma ib_device, core code creates an ib_core_device in every +net namespace to give the appearance that the rdma device is present +in all net namespaces. +Each ib_core_device owns the sysfs entries in their net namespace. + +All ib_core_device(s) points to one owner ib_device using owner pointer. + +2.1 Shared rdma ib_device view in different net namespaces +----------------------------------------------------------- + + ib_core_device (net_ns_1) + +--------------+ + | | + | device | + | +----------+ | + | | | | + | | | | + | | | | + | +----------+ | (init_net) + | *net | ib_device + | *owner-------------------------+------>+--------------------+<--+ + +--------------+ | | | | + | | ib_core_device | | + | | +--------------+ | | + | | | | | | + | | | device | | | + | | | +----------+ | | | + ib_core_device (net_ns_2) | | | | | | | | + +--------------+ | | | | | | | | + | | | | | | | | | | + | device | | | | +----------+ | | | + | +----------+ | | | | *net | | | + | | | | | | | *owner--------------+ + | | | | | | +--------------+ | + | | | | | +--------------------+ + | +----------+ | | + | *net | | + | *owner------------------------+ + +--------------+ + +2.2 rdma ib_device bound to a net namespace (in future) +-------------------------------------------------------- + +In this mode, when an rdma device is bound to a net namespace, all compat +sysfs entries will be terminated. sysfs entries will reside in single +net namespace which device is bound to. +Thereby having one-to-one mapping and providing isolation of devices +to their owning net namespace. + +(net_ns_1) +ib_device ++--------------------+ +| | +| | +| ib_core_device | +| +--------------+ | +| | | | +| | device | | +| | +----------+ | | +| | | | | | +| | | | | | +| | | | | | +| | +----------+ | | +| | | | +| | *net | | +| | *owner | | +| +--------------+ | ++--------------------+ + +2.3 locking scheme +-------------------------------------------------------- +There are three locks involved to provide synchronization between five +operations. +These five operations are +(a) device addition using ib_register_device() +(b) device removal using ib_unregister_device() +(c) net namespace addition using _init_net() notifier +(d) net namespace removal using _exit_net() notifier +(e) device renaming netlink command + +Each of above operations can happen in parallel. +Few interesting combinations to consider are: +1. init_net() and register_device() trying to add compat devices +2. exit_net() and unregister_device() trying to remove compat devices +3. renaming compat devices while doing init_net() or exit_net(). + +Net namespaces are identified using a unique id in an xarray. +This xarray operation is protected using rdma_net_rwsem. +Same id is being used for adding compat device for a given rdma device. + +compat devices of a given ib device is maintained using per device xarray. +This xarray is used because two paths - net ns notifiers and device life cycle +routines, both attempt to add compat devices. Such work is protected using per +device compat_rw_mutex. + +Below lock sequence ensures that whoever sees the device adds/removes compat +devices for a given net namespace(s). + + cpu-0 cpu-1 + ----- ----- +init_net()/exit_net() reg_dev()/unreg_dev() + + lock_N lock_D + [..] [..] + unlock_N [..] + unlock_D + + lock_N + [..] + lock_D unlock_N + [..] + unlock_D From patchwork Wed Feb 13 17:23:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 10810465 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 07A456C2 for ; Wed, 13 Feb 2019 17:23:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DCCE92DD1C for ; Wed, 13 Feb 2019 17:23:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DADE82DD81; Wed, 13 Feb 2019 17:23:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 585502DD1C for ; Wed, 13 Feb 2019 17:23:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388538AbfBMRX4 (ORCPT ); Wed, 13 Feb 2019 12:23:56 -0500 Received: from mail.kernel.org ([198.145.29.99]:53900 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726026AbfBMRX4 (ORCPT ); Wed, 13 Feb 2019 12:23:56 -0500 Received: from localhost (unknown [77.138.135.184]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A36062175B; Wed, 13 Feb 2019 17:23:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1550078635; bh=xzfcwFzH/I7GLQStdflz+ewXfig+FQI8Aud3xsnFpxk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BPbjWlRfs5i4TwpkaXjcftvyBJbOCc8k4oa29YEcimfnATHVHEEVHGTCCa4mXyWpR EZIia7hDawSi8Y3g0EiKUG9PtGmUXidOwpxS0f32ftzBWlfcSpL4IQazXw9cbJfVJO dtFazqjSa7lKveAZUo/nCG9SHYqHZtSglY5yDoDU= From: Leon Romanovsky To: Doug Ledford , Jason Gunthorpe Cc: Leon Romanovsky , RDMA mailing list , Parav Pandit Subject: [PATCH rdma-next 8/8] RDMA/core: Support core port attributes in non init_net Date: Wed, 13 Feb 2019 19:23:10 +0200 Message-Id: <20190213172310.1681-9-leon@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190213172310.1681-1-leon@kernel.org> References: <20190213172310.1681-1-leon@kernel.org> MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Parav Pandit Now that sysfs compatibility layer for non init_net exists, add core port attributes such as pkey and gid table to non init_net ns. Signed-off-by: Parav Pandit Signed-off-by: Leon Romanovsky --- drivers/infiniband/core/core_priv.h | 4 ++++ drivers/infiniband/core/device.c | 6 ++++++ drivers/infiniband/core/sysfs.c | 17 +++++++++-------- 3 files changed, 19 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h index 7705aa6861b5..7ca1440cf069 100644 --- a/drivers/infiniband/core/core_priv.h +++ b/drivers/infiniband/core/core_priv.h @@ -336,4 +336,8 @@ struct net_device *rdma_read_gid_attr_ndev_rcu(const struct ib_gid_attr *attr); void rdma_init_coredev(struct ib_core_device *coredev, struct ib_device *dev, struct net *net); + +void ib_free_port_attrs(struct ib_core_device *coredev); +int ib_setup_port_attrs(struct ib_core_device *coredev, + bool alloc_hw_stats); #endif /* _CORE_PRIV_H */ diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index 0ab28ab801af..4dc5bb2f5c77 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -715,6 +715,9 @@ static int add_one_compat_dev(struct ib_device *device, ret = device_add(&cdev->coredev.dev); if (ret) goto add_err; + ret = ib_setup_port_attrs(&cdev->coredev, false); + if (ret) + goto port_err; ret = xa_insert(&device->compat_devs, rnet->id, cdev, GFP_KERNEL); if (ret) @@ -724,6 +727,8 @@ static int add_one_compat_dev(struct ib_device *device, return 0; insert_err: + ib_free_port_attrs(&cdev->coredev); +port_err: device_del(&cdev->coredev.dev); add_err: put_device(&cdev->coredev.dev); @@ -740,6 +745,7 @@ static void remove_one_compat_dev(struct ib_device *device, u32 id) cdev = xa_erase(&device->compat_devs, id); mutex_unlock(&device->compat_devs_mutex); if (cdev) { + ib_free_port_attrs(&cdev->coredev); device_del(&cdev->coredev.dev); put_device(&cdev->coredev.dev); } diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c index 55d7d975393a..0ede5f3a5817 100644 --- a/drivers/infiniband/core/sysfs.c +++ b/drivers/infiniband/core/sysfs.c @@ -1015,7 +1015,8 @@ static void setup_hw_stats(struct ib_device *device, struct ib_port *port, return; } -static int add_port(struct ib_core_device *coredev, int port_num) +static int add_port(struct ib_core_device *coredev, + int port_num, bool alloc_stats) { struct ib_device *device = rdma_device_to_ibdev(&coredev->dev); struct ib_port *p; @@ -1056,7 +1057,7 @@ static int add_port(struct ib_core_device *coredev, int port_num) goto err_put; } - if (device->ops.process_mad) { + if (device->ops.process_mad && alloc_stats) { p->pma_table = get_counter_table(device, port_num); ret = sysfs_create_group(&p->kobj, p->pma_table); if (ret) @@ -1123,7 +1124,7 @@ static int add_port(struct ib_core_device *coredev, int port_num) * port, so holder should be device. Therefore skip per port conunter * initialization. */ - if (device->ops.alloc_hw_stats && port_num) + if (device->ops.alloc_hw_stats && port_num && alloc_stats) setup_hw_stats(device, p, port_num); list_add_tail(&p->kobj.entry, &coredev->port_list); @@ -1280,7 +1281,7 @@ const struct attribute_group ib_dev_attr_group = { .attrs = ib_dev_attrs, }; -static void ib_free_port_attrs(struct ib_core_device *coredev) +void ib_free_port_attrs(struct ib_core_device *coredev) { struct kobject *p, *t; @@ -1307,7 +1308,7 @@ static void ib_free_port_attrs(struct ib_core_device *coredev) kobject_put(coredev->ports_kobj); } -static int ib_setup_port_attrs(struct ib_core_device *coredev) +int ib_setup_port_attrs(struct ib_core_device *coredev, bool alloc_stats) { struct ib_device *device = rdma_device_to_ibdev(&coredev->dev); int ret; @@ -1319,12 +1320,12 @@ static int ib_setup_port_attrs(struct ib_core_device *coredev) return -ENOMEM; if (rdma_cap_ib_switch(device)) { - ret = add_port(coredev, 0); + ret = add_port(coredev, 0, alloc_stats); if (ret) goto err_put; } else { for (i = 1; i <= device->phys_port_cnt; ++i) { - ret = add_port(coredev, i); + ret = add_port(coredev, i, alloc_stats); if (ret) goto err_put; } @@ -1341,7 +1342,7 @@ int ib_device_register_sysfs(struct ib_device *device) { int ret; - ret = ib_setup_port_attrs(&device->coredev); + ret = ib_setup_port_attrs(&device->coredev, true); if (ret) return ret;