From patchwork Sun Oct 23 22:04:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 13016130 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 675F7ECAAA1 for ; Sun, 23 Oct 2022 05:39:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229799AbiJWFje (ORCPT ); Sun, 23 Oct 2022 01:39:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229568AbiJWFje (ORCPT ); Sun, 23 Oct 2022 01:39:34 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7920ABC9A for ; Sat, 22 Oct 2022 22:39:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666503569; x=1698039569; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=25b1xYV5TYWhqHzmUN6NIa4uHhOhs6nndlUOxy/wtLk=; b=nPIH+LqpUeDmmzO0fGXflmay+wGd6dHciZLWSKa9dJsB9Z4EG8h7M6J8 PESxrk/PR2ZxQJF7riyoYLuBCCxwfoh1iYi7mMclyd4lASbeHU6uKlqGl 1mmGXMJkzOxK2Pp3h0ZBJ4UCWxPVFR+ddWu/YdrPE1JL+GWYsUBPTl4XS 7SO+3ZcoJdIM1ytto8CoYs+YTjSSdjXUXoPbhI+t/8To612bVehn1FU/6 BkomH6atqBSVsH0fPHO70+g/8PViZpSVdKJHoaIZr7TbMiAyelNI2L6MR BUaDuWqyiiHuJ8czxTLeqO1TK7QtlejxVkO238fjhLIQ2YdGmkPoeYi0h A==; X-IronPort-AV: E=McAfee;i="6500,9779,10508"; a="294632906" X-IronPort-AV: E=Sophos;i="5.95,206,1661842800"; d="scan'208";a="294632906" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2022 22:39:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10508"; a="662047003" X-IronPort-AV: E=Sophos;i="5.95,206,1661842800"; d="scan'208";a="662047003" Received: from unknown (HELO intel-71.bj.intel.com) ([10.238.154.71]) by orsmga008.jf.intel.com with ESMTP; 22 Oct 2022 22:39:27 -0700 From: Zhu Yanjun To: jgg@ziepe.ca, leon@kernel.org, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev, yanjun.zhu@intel.com Subject: [PATCH 1/3] RDMA/core: Move ib device to the same net namespace with net device Date: Sun, 23 Oct 2022 18:04:48 -0400 Message-Id: <20221023220450.2287909-2-yanjun.zhu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20221023220450.2287909-1-yanjun.zhu@intel.com> References: <20221023220450.2287909-1-yanjun.zhu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Zhu Yanjun When a net device is moved from a net namespace to another net namespace, the related ib device should also be moved to the same net namespace. Signed-off-by: Zhu Yanjun --- drivers/infiniband/core/device.c | 57 ++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index d275db195f1a..59784fd10876 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -2735,6 +2735,54 @@ int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents) EXPORT_SYMBOL(ib_dma_virt_map_sg); #endif /* CONFIG_INFINIBAND_VIRT_DMA */ +static int rdma_netns_notify(struct notifier_block *not_blk, + unsigned long event, void *arg) +{ + struct net_device *ndev = netdev_notifier_info_to_dev(arg); + struct ib_device *ibdev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN); + + if (!ibdev) + return NOTIFY_OK; + + switch (event) { + case NETDEV_REGISTER: + ib_device_put(ibdev); + if (!net_eq(read_pnet(&ibdev->coredev.rdma_net), dev_net(ndev))) { + int ret = 0; + + get_device(&ibdev->dev); + ret = rdma_dev_change_netns(ibdev, + read_pnet(&ibdev->coredev.rdma_net), + dev_net(ndev)); + if (ret) { + put_device(&ibdev->dev); + return NOTIFY_OK; + } + put_device(&ibdev->dev); + } + break; + case NETDEV_UNREGISTER: + ib_device_put(ibdev); + break; + case NETDEV_REBOOT: + case NETDEV_GOING_DOWN: + case NETDEV_CHANGEADDR: + case NETDEV_CHANGENAME: + case NETDEV_FEAT_CHANGE: + default: + ib_device_put(ibdev); + pr_info("ignoring netdev event = %ld for %s\n", + event, ndev->name); + break; + } + + return NOTIFY_OK; +} + +static struct notifier_block rdma_netns_notifier = { + .notifier_call = rdma_netns_notify, +}; + static const struct rdma_nl_cbs ibnl_ls_cb_table[RDMA_NL_LS_NUM_OPS] = { [RDMA_NL_LS_OP_RESOLVE] = { .doit = ib_nl_handle_resolve_resp, @@ -2817,8 +2865,16 @@ static int __init ib_core_init(void) rdma_nl_register(RDMA_NL_LS, ibnl_ls_cb_table); roce_gid_mgmt_init(); + ret = register_netdevice_notifier(&rdma_netns_notifier); + if (ret) { + pr_err("Failed to register netdev notifier\n"); + goto err_netdevice; + } + return 0; +err_netdevice: + unregister_pernet_device(&rdma_dev_net_ops); err_compat: unregister_blocking_lsm_notifier(&ibdev_lsm_nb); err_sa: @@ -2842,6 +2898,7 @@ static int __init ib_core_init(void) static void __exit ib_core_cleanup(void) { + unregister_netdevice_notifier(&rdma_netns_notifier); roce_gid_mgmt_cleanup(); nldev_exit(); rdma_nl_unregister(RDMA_NL_LS); From patchwork Sun Oct 23 22:04:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 13016131 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF563C3A59D for ; Sun, 23 Oct 2022 05:39:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229568AbiJWFjf (ORCPT ); Sun, 23 Oct 2022 01:39:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229588AbiJWFje (ORCPT ); Sun, 23 Oct 2022 01:39:34 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7634613CCC for ; Sat, 22 Oct 2022 22:39:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666503570; x=1698039570; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=mESt4mhsp+XsG+gJr28pIhhED4A6IjoZZpefU6QgPTo=; b=Iwcw2+vQELMNSQWfB6uSNT6AbgcVA2zXF/yZLxeLyjVwHTXKkikTBNbI YFkZtXOFurlfy85hAy6uwi4au6dedt2kyPG4izlTMnX8BuYdX+hkVCn/B x7umDZviPIVAcYNoRhVp41E6cUshPmjqagSj3+9ZgK+DWNzH2VfaAoawX YOlyLxP8beTW+hBrtbKocaQIGOvrpF18VK4hos7fdnwVdRKSjYcuq1T8B c9IcEIzYNIZHO58mvwNAcR+lUeK1qPuyzQa69lKRjZaYf412pI96deHKW S4gwOh6dxJZSMaOP0L9+C5fSNcggdhDy7peRskbjoQbUzBKMiebfQSiya g==; X-IronPort-AV: E=McAfee;i="6500,9779,10508"; a="294632907" X-IronPort-AV: E=Sophos;i="5.95,206,1661842800"; d="scan'208";a="294632907" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2022 22:39:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10508"; a="662047006" X-IronPort-AV: E=Sophos;i="5.95,206,1661842800"; d="scan'208";a="662047006" Received: from unknown (HELO intel-71.bj.intel.com) ([10.238.154.71]) by orsmga008.jf.intel.com with ESMTP; 22 Oct 2022 22:39:28 -0700 From: Zhu Yanjun To: jgg@ziepe.ca, leon@kernel.org, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev, yanjun.zhu@intel.com Subject: [PATCH 2/3] RDMA/core: The legacy IB devices still work with shared/exclusive mode Date: Sun, 23 Oct 2022 18:04:49 -0400 Message-Id: <20221023220450.2287909-3-yanjun.zhu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20221023220450.2287909-1-yanjun.zhu@intel.com> References: <20221023220450.2287909-1-yanjun.zhu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Zhu Yanjun Other ib devices except legacy IB devices should work with its related net devices. That is, these ib devices should be in the same net namespace with the related net devices. Signed-off-by: Zhu Yanjun --- drivers/infiniband/core/device.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index 59784fd10876..d38eb1fc2ed7 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -140,8 +140,13 @@ MODULE_PARM_DESC(netns_mode, */ bool rdma_dev_access_netns(const struct ib_device *dev, const struct net *net) { - return (ib_devices_shared_netns || - net_eq(read_pnet(&dev->coredev.rdma_net), net)); + /* ib_devices_shared_netns is only for IB device. */ + if (rdma_protocol_ib(dev, rdma_start_port(dev))) { + return (ib_devices_shared_netns || + net_eq(read_pnet(&dev->coredev.rdma_net), net)); + } else { /* Others device */ + return net_eq(read_pnet(&dev->coredev.rdma_net), net); + } } EXPORT_SYMBOL(rdma_dev_access_netns); @@ -2744,6 +2749,12 @@ static int rdma_netns_notify(struct notifier_block *not_blk, if (!ibdev) return NOTIFY_OK; + /* This will exclude IB device */ + if (rdma_protocol_ib(ibdev, rdma_start_port(ibdev))) { + ib_device_put(ibdev); + return NOTIFY_OK; + } + switch (event) { case NETDEV_REGISTER: ib_device_put(ibdev); From patchwork Sun Oct 23 22:04:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhu Yanjun X-Patchwork-Id: 13016132 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8F3FECAAA1 for ; Sun, 23 Oct 2022 05:39:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229588AbiJWFjj (ORCPT ); Sun, 23 Oct 2022 01:39:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229961AbiJWFjh (ORCPT ); Sun, 23 Oct 2022 01:39:37 -0400 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39BDD15A39 for ; Sat, 22 Oct 2022 22:39:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1666503574; x=1698039574; h=from:to:subject:date:message-id:in-reply-to:references: mime-version:content-transfer-encoding; bh=k0hNa9sSqDuaDoEKGYzhyGC+62vA6BwjfQQq0pLVRfY=; b=OJsinB5uX0qV3kPs9HEkWGEgNIba7+eMwATVIGEkZw+xyyOoupEbzFAY pYk3c0lB9avoBOnVESps7dajVWOGc0xLQX1MZ39q0SSQvYbYWnaZu5mTM 5WgqYNw5ppSDo/OaSY6+Hhnl43SMONYFu3Kir8W/5bvBP7GIhyIr04vWM mZv14kUkmOHapWKJJCtv46nwAF1VvX/E/8BAtOAzxS/h/g+xGuMyIBN3I 1XmF/wiBdiz3xyoioVAIKzhw6PYjIj29RZyu2ftzHFG1YuuiZaCxLObA8 V/380HepW2vo6CEuQDU63elT6HNbUWohq5MFo2ewvYqpMK4fDDDLSU7Eb Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10508"; a="294632912" X-IronPort-AV: E=Sophos;i="5.95,206,1661842800"; d="scan'208";a="294632912" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Oct 2022 22:39:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10508"; a="662047012" X-IronPort-AV: E=Sophos;i="5.95,206,1661842800"; d="scan'208";a="662047012" Received: from unknown (HELO intel-71.bj.intel.com) ([10.238.154.71]) by orsmga008.jf.intel.com with ESMTP; 22 Oct 2022 22:39:30 -0700 From: Zhu Yanjun To: jgg@ziepe.ca, leon@kernel.org, linux-rdma@vger.kernel.org, yanjun.zhu@linux.dev, yanjun.zhu@intel.com Subject: [PATCH 3/3] RDMA/core: Get all the ib devices from net devices Date: Sun, 23 Oct 2022 18:04:50 -0400 Message-Id: <20221023220450.2287909-4-yanjun.zhu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20221023220450.2287909-1-yanjun.zhu@intel.com> References: <20221023220450.2287909-1-yanjun.zhu@intel.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Zhu Yanjun To mlx4/5, the function ib_device_get_by_netdev can not get ib devices from net devices. So this function parses all the ib devices to get all the net devices, then compares the net devices with the given net device. Signed-off-by: Zhu Yanjun --- drivers/infiniband/core/device.c | 39 ++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c index d38eb1fc2ed7..55e2d75d752e 100644 --- a/drivers/infiniband/core/device.c +++ b/drivers/infiniband/core/device.c @@ -2740,14 +2740,49 @@ int ib_dma_virt_map_sg(struct ib_device *dev, struct scatterlist *sg, int nents) EXPORT_SYMBOL(ib_dma_virt_map_sg); #endif /* CONFIG_INFINIBAND_VIRT_DMA */ +static struct ib_device *get_ibdev_from_ndev(struct net_device *ndev) +{ + unsigned long index; + struct ib_device *dev; + int i; + + down_read(&devices_rwsem); + xa_for_each_marked(&devices, index, dev, DEVICE_REGISTERED) { + if (!dev->ops.get_netdev) + continue; + + for (i = 0; i < dev->phys_port_cnt; i++) { + struct net_device *netdev; + + netdev = dev->ops.get_netdev(dev, i+1); + if (!netdev) + continue; + + dev_put(netdev); + if (ndev == netdev) { + up_read(&devices_rwsem); + if (!ib_device_try_get(dev)) + dev = NULL; + return dev; + } + } + } + up_read(&devices_rwsem); + return NULL; +} + static int rdma_netns_notify(struct notifier_block *not_blk, unsigned long event, void *arg) { struct net_device *ndev = netdev_notifier_info_to_dev(arg); struct ib_device *ibdev = ib_device_get_by_netdev(ndev, RDMA_DRIVER_UNKNOWN); - if (!ibdev) - return NOTIFY_OK; + if (!ibdev) { + /* This is for MLX4/5 */ + ibdev = get_ibdev_from_ndev(ndev); + if (!ibdev) + return NOTIFY_OK; + } /* This will exclude IB device */ if (rdma_protocol_ib(ibdev, rdma_start_port(ibdev))) {