From patchwork Mon Aug 21 13:12:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE792EE49AE for ; Mon, 21 Aug 2023 13:12:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235298AbjHUNM6 (ORCPT ); Mon, 21 Aug 2023 09:12:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235294AbjHUNM5 (ORCPT ); Mon, 21 Aug 2023 09:12:57 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB42BFC; Mon, 21 Aug 2023 06:12:47 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 8C7D121FDC; Mon, 21 Aug 2023 13:12:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623560; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mg+9Z7qbAnnpbMbp7NGUXXtkU2DN+Q6EW1LYSqeD6gg=; b=JAIPZxqAXXjGgF99lbUuqD8/vX7K0yXIPLgs6Z+lduPgXYXhNAOlqpeehRqnnnKOD7bZY5 6mmZN3xQ0auXLJqB4AhIK5pU0SL6o1hRFn3fXl15+MsH0Ds7jmswPkh5S+TBlQSNHpSjm3 anjDHpbmcR5agGcng7vEag6N7KZlrF8= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 47FBD139C6; Mon, 21 Aug 2023 13:12:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id UMfDEMhi42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:40 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v3 01/11] mlx4: Get rid of the mlx4_interface.get_dev callback Date: Mon, 21 Aug 2023 15:12:15 +0200 Message-Id: <20230821131225.11290-2-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Simplify the mlx4 driver interface by removing mlx4_get_protocol_dev() and the associated mlx4_interface.get_dev callbacks. This is done in preparation to use an auxiliary bus to model the mlx4 driver structure. The change is motivated by the following situation: * The mlx4_en interface is being initialized by mlx4_en_add() and mlx4_en_activate(). * The latter activate function calls mlx4_en_init_netdev() -> register_netdev() to register a new net_device. * A netdev event NETDEV_REGISTER is raised for the device. * The netdev notififier mlx4_ib_netdev_event() is called and it invokes mlx4_ib_scan_netdevs() -> mlx4_get_protocol_dev() -> mlx4_en_get_netdev() [via mlx4_interface.get_dev]. This chain creates a problem when mlx4_en gets switched to be an auxiliary driver. It contains two device calls which would both need to take a respective device lock. Avoid this situation by updating mlx4_ib_scan_netdevs() to no longer call mlx4_get_protocol_dev() but instead to utilize the information passed in net_device.parent and net_device.dev_port. This data is sufficient to determine that an updated port is one that the mlx4_ib driver should take care of and to keep mlx4_ib_dev.iboe.netdevs up to date. Following that, update mlx4_ib_get_netdev() to also not call mlx4_get_protocol_dev() and instead scan all current netdevs to find find a matching one. Note that mlx4_ib_get_netdev() is called early from ib_register_device() and cannot use data tracked in mlx4_ib_dev.iboe.netdevs which is not at that point yet set. Finally, remove function mlx4_get_protocol_dev() and the mlx4_interface.get_dev callbacks (only mlx4_en_get_netdev()) as they became unused. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/infiniband/hw/mlx4/main.c | 89 ++++++++++---------- drivers/net/ethernet/mellanox/mlx4/en_main.c | 8 -- drivers/net/ethernet/mellanox/mlx4/intf.c | 21 ----- include/linux/mlx4/driver.h | 3 - 4 files changed, 43 insertions(+), 78 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index b18e9f2adc82..7dd70d778b6b 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -125,12 +125,14 @@ static struct net_device *mlx4_ib_get_netdev(struct ib_device *device, u32 port_num) { struct mlx4_ib_dev *ibdev = to_mdev(device); - struct net_device *dev; + struct net_device *dev, *ret = NULL; rcu_read_lock(); - dev = mlx4_get_protocol_dev(ibdev->dev, MLX4_PROT_ETH, port_num); + for_each_netdev_rcu(&init_net, dev) { + if (dev->dev.parent != ibdev->ib_dev.dev.parent || + dev->dev_port + 1 != port_num) + continue; - if (dev) { if (mlx4_is_bonded(ibdev->dev)) { struct net_device *upper = NULL; @@ -143,11 +145,14 @@ static struct net_device *mlx4_ib_get_netdev(struct ib_device *device, dev = active; } } + + dev_hold(dev); + ret = dev; + break; } - dev_hold(dev); rcu_read_unlock(); - return dev; + return ret; } static int mlx4_ib_update_gids_v1(struct gid_entry *gids, @@ -2319,61 +2324,53 @@ static void mlx4_ib_update_qps(struct mlx4_ib_dev *ibdev, mutex_unlock(&ibdev->qp1_proxy_lock[port - 1]); } -static void mlx4_ib_scan_netdevs(struct mlx4_ib_dev *ibdev, - struct net_device *dev, - unsigned long event) +static void mlx4_ib_scan_netdev(struct mlx4_ib_dev *ibdev, + struct net_device *dev, + unsigned long event) { - struct mlx4_ib_iboe *iboe; - int update_qps_port = -1; - int port; + struct mlx4_ib_iboe *iboe = &ibdev->iboe; ASSERT_RTNL(); - iboe = &ibdev->iboe; + if (dev->dev.parent != ibdev->ib_dev.dev.parent) + return; spin_lock_bh(&iboe->lock); - mlx4_foreach_ib_transport_port(port, ibdev->dev) { - - iboe->netdevs[port - 1] = - mlx4_get_protocol_dev(ibdev->dev, MLX4_PROT_ETH, port); - if (dev == iboe->netdevs[port - 1] && - (event == NETDEV_CHANGEADDR || event == NETDEV_REGISTER || - event == NETDEV_UP || event == NETDEV_CHANGE)) - update_qps_port = port; + iboe->netdevs[dev->dev_port] = event != NETDEV_UNREGISTER ? dev : NULL; - if (dev == iboe->netdevs[port - 1] && - (event == NETDEV_UP || event == NETDEV_DOWN)) { - enum ib_port_state port_state; - struct ib_event ibev = { }; - - if (ib_get_cached_port_state(&ibdev->ib_dev, port, - &port_state)) - continue; + if (event == NETDEV_UP || event == NETDEV_DOWN) { + enum ib_port_state port_state; + struct ib_event ibev = { }; - if (event == NETDEV_UP && - (port_state != IB_PORT_ACTIVE || - iboe->last_port_state[port - 1] != IB_PORT_DOWN)) - continue; - if (event == NETDEV_DOWN && - (port_state != IB_PORT_DOWN || - iboe->last_port_state[port - 1] != IB_PORT_ACTIVE)) - continue; - iboe->last_port_state[port - 1] = port_state; + if (ib_get_cached_port_state(&ibdev->ib_dev, dev->dev_port + 1, + &port_state)) + goto iboe_out; - ibev.device = &ibdev->ib_dev; - ibev.element.port_num = port; - ibev.event = event == NETDEV_UP ? IB_EVENT_PORT_ACTIVE : - IB_EVENT_PORT_ERR; - ib_dispatch_event(&ibev); - } + if (event == NETDEV_UP && + (port_state != IB_PORT_ACTIVE || + iboe->last_port_state[dev->dev_port] != IB_PORT_DOWN)) + goto iboe_out; + if (event == NETDEV_DOWN && + (port_state != IB_PORT_DOWN || + iboe->last_port_state[dev->dev_port] != IB_PORT_ACTIVE)) + goto iboe_out; + iboe->last_port_state[dev->dev_port] = port_state; + ibev.device = &ibdev->ib_dev; + ibev.element.port_num = dev->dev_port + 1; + ibev.event = event == NETDEV_UP ? IB_EVENT_PORT_ACTIVE : + IB_EVENT_PORT_ERR; + ib_dispatch_event(&ibev); } + +iboe_out: spin_unlock_bh(&iboe->lock); - if (update_qps_port > 0) - mlx4_ib_update_qps(ibdev, dev, update_qps_port); + if (event == NETDEV_CHANGEADDR || event == NETDEV_REGISTER || + event == NETDEV_UP || event == NETDEV_CHANGE) + mlx4_ib_update_qps(ibdev, dev, dev->dev_port + 1); } static int mlx4_ib_netdev_event(struct notifier_block *this, @@ -2386,7 +2383,7 @@ static int mlx4_ib_netdev_event(struct notifier_block *this, return NOTIFY_DONE; ibdev = container_of(this, struct mlx4_ib_dev, iboe.nb); - mlx4_ib_scan_netdevs(ibdev, dev, event); + mlx4_ib_scan_netdev(ibdev, dev, event); return NOTIFY_DONE; } diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index f1259bdb1a29..6a42bec6bd85 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -183,13 +183,6 @@ static void mlx4_en_get_profile(struct mlx4_en_dev *mdev) } } -static void *mlx4_en_get_netdev(struct mlx4_dev *dev, void *ctx, u8 port) -{ - struct mlx4_en_dev *endev = ctx; - - return endev->pndev[port]; -} - static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, enum mlx4_dev_event event, unsigned long port) { @@ -354,7 +347,6 @@ static struct mlx4_interface mlx4_en_interface = { .add = mlx4_en_add, .remove = mlx4_en_remove, .event = mlx4_en_event, - .get_dev = mlx4_en_get_netdev, .protocol = MLX4_PROT_ETH, .activate = mlx4_en_activate, }; diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 65482f004e50..28d7da925d36 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -245,27 +245,6 @@ void mlx4_unregister_device(struct mlx4_dev *dev) mutex_unlock(&intf_mutex); } -void *mlx4_get_protocol_dev(struct mlx4_dev *dev, enum mlx4_protocol proto, int port) -{ - struct mlx4_priv *priv = mlx4_priv(dev); - struct mlx4_device_context *dev_ctx; - unsigned long flags; - void *result = NULL; - - spin_lock_irqsave(&priv->ctx_lock, flags); - - list_for_each_entry(dev_ctx, &priv->ctx_list, list) - if (dev_ctx->intf->protocol == proto && dev_ctx->intf->get_dev) { - result = dev_ctx->intf->get_dev(dev, dev_ctx->context, port); - break; - } - - spin_unlock_irqrestore(&priv->ctx_lock, flags); - - return result; -} -EXPORT_SYMBOL_GPL(mlx4_get_protocol_dev); - struct devlink_port *mlx4_get_devlink_port(struct mlx4_dev *dev, int port) { struct mlx4_port_info *info = &mlx4_priv(dev)->port[port]; diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 1834c8fad12e..923951e19300 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -59,7 +59,6 @@ struct mlx4_interface { void (*remove)(struct mlx4_dev *dev, void *context); void (*event) (struct mlx4_dev *dev, void *context, enum mlx4_dev_event event, unsigned long param); - void * (*get_dev)(struct mlx4_dev *dev, void *context, u8 port); void (*activate)(struct mlx4_dev *dev, void *context); struct list_head list; enum mlx4_protocol protocol; @@ -88,8 +87,6 @@ struct mlx4_port_map { int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p); -void *mlx4_get_protocol_dev(struct mlx4_dev *dev, enum mlx4_protocol proto, int port); - struct devlink_port *mlx4_get_devlink_port(struct mlx4_dev *dev, int port); #endif /* MLX4_DRIVER_H */ From patchwork Mon Aug 21 13:12:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BA71EE4993 for ; Mon, 21 Aug 2023 13:12:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232017AbjHUNMt (ORCPT ); Mon, 21 Aug 2023 09:12:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230435AbjHUNMt (ORCPT ); Mon, 21 Aug 2023 09:12:49 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51F3BEB; Mon, 21 Aug 2023 06:12:42 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id D7B3B206BD; Mon, 21 Aug 2023 13:12:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623560; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MlkLwPoGSJ/eBik/CH7trGhSdIICH1/nZbIFJG2ENWw=; b=GImSM49SZORkz8eK5AfccoOjASdmb/KHBqijohv8FfjLdSOuVxtJiYneqzdqNtdK1I7rio BrSNhNmXLExsCik0O7jLd+YLEamKUscOhCYvH/P01f6C2dgOKFkLkn+8RqsJV1xrlIpfJl dC5c9fHY94pP5gb9/PyzSWKF+xSlong= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 909B513421; Mon, 21 Aug 2023 13:12:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 4CSDIshi42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:40 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v3 02/11] mlx4: Rename member mlx4_en_dev.nb to netdev_nb Date: Mon, 21 Aug 2023 15:12:16 +0200 Message-Id: <20230821131225.11290-3-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Rename the mlx4_en_dev.nb notifier_block member to netdev_nb in preparation to add a mlx4 core notifier_block. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/en_main.c | 14 +++++++------- drivers/net/ethernet/mellanox/mlx4/en_netdev.c | 2 +- drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 2 +- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index 6a42bec6bd85..be8ba34c9025 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -235,8 +235,8 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) iounmap(mdev->uar_map); mlx4_uar_free(dev, &mdev->priv_uar); mlx4_pd_free(dev, mdev->priv_pdn); - if (mdev->nb.notifier_call) - unregister_netdevice_notifier(&mdev->nb); + if (mdev->netdev_nb.notifier_call) + unregister_netdevice_notifier(&mdev->netdev_nb); kfree(mdev); } @@ -252,11 +252,11 @@ static void mlx4_en_activate(struct mlx4_dev *dev, void *ctx) mdev->pndev[i] = NULL; } - /* register notifier */ - mdev->nb.notifier_call = mlx4_en_netdev_event; - if (register_netdevice_notifier(&mdev->nb)) { - mdev->nb.notifier_call = NULL; - mlx4_err(mdev, "Failed to create notifier\n"); + /* register netdev notifier */ + mdev->netdev_nb.notifier_call = mlx4_en_netdev_event; + if (register_netdevice_notifier(&mdev->netdev_nb)) { + mdev->netdev_nb.notifier_call = NULL; + mlx4_err(mdev, "Failed to create netdev notifier\n"); } } diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c index 403604ceebc8..7066c426b95c 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c @@ -2967,7 +2967,7 @@ int mlx4_en_netdev_event(struct notifier_block *this, if (!net_eq(dev_net(ndev), &init_net)) return NOTIFY_DONE; - mdev = container_of(this, struct mlx4_en_dev, nb); + mdev = container_of(this, struct mlx4_en_dev, netdev_nb); dev = mdev->dev; /* Go into this mode only when two network devices set on two ports diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h index 321f801c1d7c..72a3fea36702 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h @@ -432,7 +432,7 @@ struct mlx4_en_dev { unsigned long last_overflow_check; struct ptp_clock *ptp_clock; struct ptp_clock_info ptp_clock_info; - struct notifier_block nb; + struct notifier_block netdev_nb; }; From patchwork Mon Aug 21 13:12:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06188EE49AA for ; Mon, 21 Aug 2023 13:12:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230435AbjHUNMu (ORCPT ); Mon, 21 Aug 2023 09:12:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235257AbjHUNMt (ORCPT ); Mon, 21 Aug 2023 09:12:49 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95867EC; Mon, 21 Aug 2023 06:12:42 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 2A898206BC; Mon, 21 Aug 2023 13:12:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623561; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qC3K0GntmmvyP3SrhZ2T0PGnlm8nlXrEXp8klJKww5Y=; b=eJ6keGmk7KFTwVWC0WrQXvCLMbdFvCEquWf8JquFX5hkXP/OkWKMzPfhYxBtUoKyFHuhiQ c63h+7kf6KJ3Rf82jkPuJxKw1HJWVbkLmDXG+Ckbu5eMr0LWMTm+NWk65nEbGQP6le6pNs DJ3LSqpzcfNfcaqxjFRJ4O21VX6u/1s= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id DD232139BC; Mon, 21 Aug 2023 13:12:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 6P4kNchi42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:40 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu Subject: [PATCH net-next v3 03/11] mlx4: Use 'void *' as the event param of mlx4_dispatch_event() Date: Mon, 21 Aug 2023 15:12:17 +0200 Message-Id: <20230821131225.11290-4-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Function mlx4_dispatch_event() takes an 'unsigned long' as its event parameter. The actual value is none (MLX4_DEV_EVENT_CATASTROPHIC_ERROR), a pointer to mlx4_eqe (MLX4_DEV_EVENT_PORT_MGMT_CHANGE), or a 32-bit integer (remaining events). In preparation to switch mlx4_en and mlx4_ib to be an auxiliary device, the mlx4_interface.event callback is replaced with a notifier and function mlx4_dispatch_event() gets updated to invoke atomic_notifier_call_chain(). This requires forwarding the input 'param' value from the former function to the latter. A problem is that the notifier call takes 'void *' as its 'param' value, compared to 'unsigned long' used by mlx4_dispatch_event(). Re-passing the value would need either punning it to 'void *' or passing down the address of the input 'param'. Both approaches create a number of unnecessary casts. Change instead the input 'param' of mlx4_dispatch_event() from 'unsigned long' to 'void *'. A mlx4_eqe pointer can be passed directly, callers using an int value are adjusted to pass its address. Signed-off-by: Petr Pavlu Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/mlx4/main.c | 14 ++++++++++---- drivers/net/ethernet/mellanox/mlx4/catas.c | 2 +- drivers/net/ethernet/mellanox/mlx4/cmd.c | 4 ++-- drivers/net/ethernet/mellanox/mlx4/en_main.c | 17 +++++++++++++++-- drivers/net/ethernet/mellanox/mlx4/eq.c | 15 ++++++++------- drivers/net/ethernet/mellanox/mlx4/intf.c | 2 +- drivers/net/ethernet/mellanox/mlx4/mlx4.h | 2 +- include/linux/mlx4/driver.h | 2 +- 8 files changed, 39 insertions(+), 19 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 7dd70d778b6b..2c5fd8174b3c 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -3174,7 +3174,7 @@ void mlx4_sched_ib_sl2vl_update_work(struct mlx4_ib_dev *ibdev, } static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, - enum mlx4_dev_event event, unsigned long param) + enum mlx4_dev_event event, void *param) { struct ib_event ibev; struct mlx4_ib_dev *ibdev = to_mdev((struct ib_device *) ibdev_ptr); @@ -3194,10 +3194,16 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, return; } - if (event == MLX4_DEV_EVENT_PORT_MGMT_CHANGE) + switch (event) { + case MLX4_DEV_EVENT_CATASTROPHIC_ERROR: + break; + case MLX4_DEV_EVENT_PORT_MGMT_CHANGE: eqe = (struct mlx4_eqe *)param; - else - p = (int) param; + break; + default: + p = *(int *)param; + break; + } switch (event) { case MLX4_DEV_EVENT_PORT_UP: diff --git a/drivers/net/ethernet/mellanox/mlx4/catas.c b/drivers/net/ethernet/mellanox/mlx4/catas.c index 0eb7b83637d8..0d8a362c2673 100644 --- a/drivers/net/ethernet/mellanox/mlx4/catas.c +++ b/drivers/net/ethernet/mellanox/mlx4/catas.c @@ -194,7 +194,7 @@ void mlx4_enter_error_state(struct mlx4_dev_persistent *persist) mutex_unlock(&persist->device_state_mutex); /* At that step HW was already reset, now notify clients */ - mlx4_dispatch_event(dev, MLX4_DEV_EVENT_CATASTROPHIC_ERROR, 0); + mlx4_dispatch_event(dev, MLX4_DEV_EVENT_CATASTROPHIC_ERROR, NULL); mlx4_cmd_wake_completions(dev); return; diff --git a/drivers/net/ethernet/mellanox/mlx4/cmd.c b/drivers/net/ethernet/mellanox/mlx4/cmd.c index c56d2194cbfc..f5b1f8c7834f 100644 --- a/drivers/net/ethernet/mellanox/mlx4/cmd.c +++ b/drivers/net/ethernet/mellanox/mlx4/cmd.c @@ -2113,7 +2113,7 @@ static void mlx4_master_do_cmd(struct mlx4_dev *dev, int slave, u8 cmd, if (MLX4_COMM_CMD_FLR == slave_state[slave].last_cmd) goto inform_slave_state; - mlx4_dispatch_event(dev, MLX4_DEV_EVENT_SLAVE_SHUTDOWN, slave); + mlx4_dispatch_event(dev, MLX4_DEV_EVENT_SLAVE_SHUTDOWN, &slave); /* write the version in the event field */ reply |= mlx4_comm_get_version(); @@ -2152,7 +2152,7 @@ static void mlx4_master_do_cmd(struct mlx4_dev *dev, int slave, u8 cmd, if (mlx4_master_activate_admin_state(priv, slave)) goto reset_slave; slave_state[slave].active = true; - mlx4_dispatch_event(dev, MLX4_DEV_EVENT_SLAVE_INIT, slave); + mlx4_dispatch_event(dev, MLX4_DEV_EVENT_SLAVE_INIT, &slave); break; case MLX4_COMM_CMD_VHCR_POST: if ((slave_state[slave].last_cmd != MLX4_COMM_CMD_VHCR_EN) && diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index be8ba34c9025..83dae886ade6 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -184,10 +184,22 @@ static void mlx4_en_get_profile(struct mlx4_en_dev *mdev) } static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, - enum mlx4_dev_event event, unsigned long port) + enum mlx4_dev_event event, void *param) { struct mlx4_en_dev *mdev = (struct mlx4_en_dev *) endev_ptr; struct mlx4_en_priv *priv; + int port; + + switch (event) { + case MLX4_DEV_EVENT_CATASTROPHIC_ERROR: + case MLX4_DEV_EVENT_PORT_MGMT_CHANGE: + case MLX4_DEV_EVENT_SLAVE_INIT: + case MLX4_DEV_EVENT_SLAVE_SHUTDOWN: + break; + default: + port = *(int *)param; + break; + } switch (event) { case MLX4_DEV_EVENT_PORT_UP: @@ -205,6 +217,7 @@ static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, mlx4_err(mdev, "Internal error detected, restarting device\n"); break; + case MLX4_DEV_EVENT_PORT_MGMT_CHANGE: case MLX4_DEV_EVENT_SLAVE_INIT: case MLX4_DEV_EVENT_SLAVE_SHUTDOWN: break; @@ -213,7 +226,7 @@ static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, !mdev->pndev[port]) return; mlx4_warn(mdev, "Unhandled event %d for port %d\n", event, - (int) port); + port); } } diff --git a/drivers/net/ethernet/mellanox/mlx4/eq.c b/drivers/net/ethernet/mellanox/mlx4/eq.c index 414e390e6b48..6598b10a9ff4 100644 --- a/drivers/net/ethernet/mellanox/mlx4/eq.c +++ b/drivers/net/ethernet/mellanox/mlx4/eq.c @@ -501,7 +501,7 @@ static int mlx4_eq_int(struct mlx4_dev *dev, struct mlx4_eq *eq) int port; int slave = 0; int ret; - u32 flr_slave; + int flr_slave; u8 update_slave_state; int i; enum slave_port_gen_event gen_event; @@ -606,8 +606,8 @@ static int mlx4_eq_int(struct mlx4_dev *dev, struct mlx4_eq *eq) port = be32_to_cpu(eqe->event.port_change.port) >> 28; slaves_port = mlx4_phys_to_slaves_pport(dev, port); if (eqe->subtype == MLX4_PORT_CHANGE_SUBTYPE_DOWN) { - mlx4_dispatch_event(dev, MLX4_DEV_EVENT_PORT_DOWN, - port); + mlx4_dispatch_event( + dev, MLX4_DEV_EVENT_PORT_DOWN, &port); mlx4_priv(dev)->sense.do_sense_port[port] = 1; if (!mlx4_is_master(dev)) break; @@ -647,7 +647,8 @@ static int mlx4_eq_int(struct mlx4_dev *dev, struct mlx4_eq *eq) } } } else { - mlx4_dispatch_event(dev, MLX4_DEV_EVENT_PORT_UP, port); + mlx4_dispatch_event(dev, MLX4_DEV_EVENT_PORT_UP, + &port); mlx4_priv(dev)->sense.do_sense_port[port] = 0; @@ -758,7 +759,7 @@ static int mlx4_eq_int(struct mlx4_dev *dev, struct mlx4_eq *eq) } spin_unlock_irqrestore(&priv->mfunc.master.slave_state_lock, flags); mlx4_dispatch_event(dev, MLX4_DEV_EVENT_SLAVE_SHUTDOWN, - flr_slave); + &flr_slave); queue_work(priv->mfunc.master.comm_wq, &priv->mfunc.master.slave_flr_event_work); break; @@ -787,8 +788,8 @@ static int mlx4_eq_int(struct mlx4_dev *dev, struct mlx4_eq *eq) break; case MLX4_EVENT_TYPE_PORT_MNG_CHG_EVENT: - mlx4_dispatch_event(dev, MLX4_DEV_EVENT_PORT_MGMT_CHANGE, - (unsigned long) eqe); + mlx4_dispatch_event( + dev, MLX4_DEV_EVENT_PORT_MGMT_CHANGE, eqe); break; case MLX4_EVENT_TYPE_RECOVERABLE_ERROR_EVENT: diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 28d7da925d36..a761971cd0c4 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -180,7 +180,7 @@ int mlx4_do_bond(struct mlx4_dev *dev, bool enable) } void mlx4_dispatch_event(struct mlx4_dev *dev, enum mlx4_dev_event type, - unsigned long param) + void *param) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_device_context *dev_ctx; diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index 6ccf340660d9..de5699a4ddaa 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -1048,7 +1048,7 @@ int mlx4_restart_one(struct pci_dev *pdev); int mlx4_register_device(struct mlx4_dev *dev); void mlx4_unregister_device(struct mlx4_dev *dev); void mlx4_dispatch_event(struct mlx4_dev *dev, enum mlx4_dev_event type, - unsigned long param); + void *param); struct mlx4_dev_cap; struct mlx4_init_hca_param; diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 923951e19300..032d7f5bfef6 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -58,7 +58,7 @@ struct mlx4_interface { void * (*add) (struct mlx4_dev *dev); void (*remove)(struct mlx4_dev *dev, void *context); void (*event) (struct mlx4_dev *dev, void *context, - enum mlx4_dev_event event, unsigned long param); + enum mlx4_dev_event event, void *param); void (*activate)(struct mlx4_dev *dev, void *context); struct list_head list; enum mlx4_protocol protocol; From patchwork Mon Aug 21 13:12:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359432 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD241EE4996 for ; Mon, 21 Aug 2023 13:12:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235276AbjHUNM5 (ORCPT ); Mon, 21 Aug 2023 09:12:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235281AbjHUNMw (ORCPT ); Mon, 21 Aug 2023 09:12:52 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6051F2; Mon, 21 Aug 2023 06:12:42 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7E6FD206BE; Mon, 21 Aug 2023 13:12:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623561; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mX+Fj1ZShaJzTUnsc9b91eDKyRcsgzYM8HpKYZekB9k=; b=qTrLOTauvozSh0OJLygVIL64KR0GEE59CXotsv1z8cIS5WkHNHq6BDOozybWAHoaLW/qCe NxIPhiNY0DEcZ71asSjWYiufhihPE5DFppk3t0R1DJ8Sh/vZWhIhP5YswZdPToJoqnXWQ9 myGF6OfQxGnNuDUH20YEOoYEaUR8+gA= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 325A613421; Mon, 21 Aug 2023 13:12:41 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id iGZCC8li42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:41 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v3 04/11] mlx4: Replace the mlx4_interface.event callback with a notifier Date: Mon, 21 Aug 2023 15:12:18 +0200 Message-Id: <20230821131225.11290-5-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Use a notifier to implement mlx4_dispatch_event() in preparation to switch mlx4_en and mlx4_ib to be an auxiliary device. A problem is that if the mlx4_interface.event callback was replaced with something as mlx4_adrv.event then the implementation of mlx4_dispatch_event() would need to acquire a lock on a given device before executing this callback. That is necessary because otherwise there is no guarantee that the associated driver cannot get unbound when the callback is running. However, taking this lock is not possible because mlx4_dispatch_event() can be invoked from the hardirq context. Using an atomic notifier allows the driver to accurately record when it wants to receive these events and solves this problem. A handler registration is done by both mlx4_en and mlx4_ib at the end of their mlx4_interface.add callback. This matches the current situation when mlx4_add_device() would enable events for a given device immediately after this callback, by adding the device on the mlx4_priv.list. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Acked-by: Tariq Toukan Reviewed-by: Leon Romanovsky --- drivers/infiniband/hw/mlx4/main.c | 40 +++++++++++++------- drivers/infiniband/hw/mlx4/mlx4_ib.h | 2 + drivers/net/ethernet/mellanox/mlx4/en_main.c | 26 +++++++++---- drivers/net/ethernet/mellanox/mlx4/intf.c | 24 ++++++++---- drivers/net/ethernet/mellanox/mlx4/main.c | 2 + drivers/net/ethernet/mellanox/mlx4/mlx4.h | 2 + drivers/net/ethernet/mellanox/mlx4/mlx4_en.h | 2 + include/linux/mlx4/driver.h | 8 +++- 8 files changed, 75 insertions(+), 31 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 2c5fd8174b3c..6debee494a16 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -82,6 +82,8 @@ static const char mlx4_ib_version[] = static void do_slave_init(struct mlx4_ib_dev *ibdev, int slave, int do_init); static enum rdma_link_layer mlx4_ib_port_link_layer(struct ib_device *device, u32 port_num); +static int mlx4_ib_event(struct notifier_block *this, unsigned long event, + void *param); static struct workqueue_struct *wq; @@ -2836,6 +2838,12 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) do_slave_init(ibdev, j, 1); } } + + /* register mlx4 core notifier */ + ibdev->mlx_nb.notifier_call = mlx4_ib_event; + err = mlx4_register_event_notifier(dev, &ibdev->mlx_nb); + WARN(err, "failed to register mlx4 event notifier (%d)", err); + return ibdev; err_notif: @@ -2953,6 +2961,8 @@ static void mlx4_ib_remove(struct mlx4_dev *dev, void *ibdev_ptr) int p; int i; + mlx4_unregister_event_notifier(dev, &ibdev->mlx_nb); + mlx4_foreach_port(i, dev, MLX4_PORT_TYPE_IB) devlink_port_type_clear(mlx4_get_devlink_port(dev, i)); ibdev->ib_active = false; @@ -3173,11 +3183,13 @@ void mlx4_sched_ib_sl2vl_update_work(struct mlx4_ib_dev *ibdev, } } -static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, - enum mlx4_dev_event event, void *param) +static int mlx4_ib_event(struct notifier_block *this, unsigned long event, + void *param) { + struct mlx4_ib_dev *ibdev = + container_of(this, struct mlx4_ib_dev, mlx_nb); + struct mlx4_dev *dev = ibdev->dev; struct ib_event ibev; - struct mlx4_ib_dev *ibdev = to_mdev((struct ib_device *) ibdev_ptr); struct mlx4_eqe *eqe = NULL; struct ib_event_work *ew; int p = 0; @@ -3187,11 +3199,11 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, (event == MLX4_DEV_EVENT_PORT_DOWN))) { ew = kmalloc(sizeof(*ew), GFP_ATOMIC); if (!ew) - return; + return NOTIFY_DONE; INIT_WORK(&ew->work, handle_bonded_port_state_event); ew->ib_dev = ibdev; queue_work(wq, &ew->work); - return; + return NOTIFY_DONE; } switch (event) { @@ -3208,7 +3220,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, switch (event) { case MLX4_DEV_EVENT_PORT_UP: if (p > ibdev->num_ports) - return; + return NOTIFY_DONE; if (!mlx4_is_slave(dev) && rdma_port_get_link_layer(&ibdev->ib_dev, p) == IB_LINK_LAYER_INFINIBAND) { @@ -3223,7 +3235,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, case MLX4_DEV_EVENT_PORT_DOWN: if (p > ibdev->num_ports) - return; + return NOTIFY_DONE; ibev.event = IB_EVENT_PORT_ERR; break; @@ -3236,7 +3248,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, case MLX4_DEV_EVENT_PORT_MGMT_CHANGE: ew = kmalloc(sizeof *ew, GFP_ATOMIC); if (!ew) - return; + return NOTIFY_DONE; INIT_WORK(&ew->work, handle_port_mgmt_change_event); memcpy(&ew->ib_eqe, eqe, sizeof *eqe); @@ -3246,7 +3258,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, queue_work(wq, &ew->work); else handle_port_mgmt_change_event(&ew->work); - return; + return NOTIFY_DONE; case MLX4_DEV_EVENT_SLAVE_INIT: /* here, p is the slave id */ @@ -3262,7 +3274,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, 1); } } - return; + return NOTIFY_DONE; case MLX4_DEV_EVENT_SLAVE_SHUTDOWN: if (mlx4_is_master(dev)) { @@ -3278,22 +3290,22 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr, } /* here, p is the slave id */ do_slave_init(ibdev, p, 0); - return; + return NOTIFY_DONE; default: - return; + return NOTIFY_DONE; } - ibev.device = ibdev_ptr; + ibev.device = &ibdev->ib_dev; ibev.element.port_num = mlx4_is_bonded(ibdev->dev) ? 1 : (u8)p; ib_dispatch_event(&ibev); + return NOTIFY_DONE; } static struct mlx4_interface mlx4_ib_interface = { .add = mlx4_ib_add, .remove = mlx4_ib_remove, - .event = mlx4_ib_event, .protocol = MLX4_PROT_IB_IPV6, .flags = MLX4_INTFF_BONDING }; diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h index 17fee1e73a45..41ca1114a995 100644 --- a/drivers/infiniband/hw/mlx4/mlx4_ib.h +++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -644,6 +645,7 @@ struct mlx4_ib_dev { spinlock_t reset_flow_resource_lock; struct list_head qp_list; struct mlx4_ib_diag_counters diag_counters[MLX4_DIAG_COUNTERS_TYPES]; + struct notifier_block mlx_nb; }; struct ib_event_work { diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index 83dae886ade6..31bf625b8158 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -183,10 +183,12 @@ static void mlx4_en_get_profile(struct mlx4_en_dev *mdev) } } -static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, - enum mlx4_dev_event event, void *param) +static int mlx4_en_event(struct notifier_block *this, unsigned long event, + void *param) { - struct mlx4_en_dev *mdev = (struct mlx4_en_dev *) endev_ptr; + struct mlx4_en_dev *mdev = + container_of(this, struct mlx4_en_dev, mlx_nb); + struct mlx4_dev *dev = mdev->dev; struct mlx4_en_priv *priv; int port; @@ -205,7 +207,7 @@ static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, case MLX4_DEV_EVENT_PORT_UP: case MLX4_DEV_EVENT_PORT_DOWN: if (!mdev->pndev[port]) - return; + return NOTIFY_DONE; priv = netdev_priv(mdev->pndev[port]); /* To prevent races, we poll the link state in a separate task rather than changing it here */ @@ -224,10 +226,12 @@ static void mlx4_en_event(struct mlx4_dev *dev, void *endev_ptr, default: if (port < 1 || port > dev->caps.num_ports || !mdev->pndev[port]) - return; - mlx4_warn(mdev, "Unhandled event %d for port %d\n", event, + return NOTIFY_DONE; + mlx4_warn(mdev, "Unhandled event %d for port %d\n", (int)event, port); } + + return NOTIFY_DONE; } static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) @@ -235,6 +239,8 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) struct mlx4_en_dev *mdev = endev_ptr; int i; + mlx4_unregister_event_notifier(dev, &mdev->mlx_nb); + mutex_lock(&mdev->state_lock); mdev->device_up = false; mutex_unlock(&mdev->state_lock); @@ -276,7 +282,7 @@ static void mlx4_en_activate(struct mlx4_dev *dev, void *ctx) static void *mlx4_en_add(struct mlx4_dev *dev) { struct mlx4_en_dev *mdev; - int i; + int err, i; printk_once(KERN_INFO "%s", mlx4_en_version); @@ -339,6 +345,11 @@ static void *mlx4_en_add(struct mlx4_dev *dev) mutex_init(&mdev->state_lock); mdev->device_up = true; + /* register mlx4 core notifier */ + mdev->mlx_nb.notifier_call = mlx4_en_event; + err = mlx4_register_event_notifier(dev, &mdev->mlx_nb); + WARN(err, "failed to register mlx4 event notifier (%d)", err); + return mdev; err_mr: @@ -359,7 +370,6 @@ static void *mlx4_en_add(struct mlx4_dev *dev) static struct mlx4_interface mlx4_en_interface = { .add = mlx4_en_add, .remove = mlx4_en_remove, - .event = mlx4_en_event, .protocol = MLX4_PROT_ETH, .activate = mlx4_en_activate, }; diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index a761971cd0c4..fecb63e69607 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -183,17 +183,27 @@ void mlx4_dispatch_event(struct mlx4_dev *dev, enum mlx4_dev_event type, void *param) { struct mlx4_priv *priv = mlx4_priv(dev); - struct mlx4_device_context *dev_ctx; - unsigned long flags; - spin_lock_irqsave(&priv->ctx_lock, flags); + atomic_notifier_call_chain(&priv->event_nh, type, param); +} - list_for_each_entry(dev_ctx, &priv->ctx_list, list) - if (dev_ctx->intf->event) - dev_ctx->intf->event(dev, dev_ctx->context, type, param); +int mlx4_register_event_notifier(struct mlx4_dev *dev, + struct notifier_block *nb) +{ + struct mlx4_priv *priv = mlx4_priv(dev); - spin_unlock_irqrestore(&priv->ctx_lock, flags); + return atomic_notifier_chain_register(&priv->event_nh, nb); +} +EXPORT_SYMBOL(mlx4_register_event_notifier); + +int mlx4_unregister_event_notifier(struct mlx4_dev *dev, + struct notifier_block *nb) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + return atomic_notifier_chain_unregister(&priv->event_nh, nb); } +EXPORT_SYMBOL(mlx4_unregister_event_notifier); int mlx4_register_device(struct mlx4_dev *dev) { diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index 8a5409b00530..5f3ba8385e23 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -3378,6 +3378,8 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, INIT_LIST_HEAD(&priv->ctx_list); spin_lock_init(&priv->ctx_lock); + ATOMIC_INIT_NOTIFIER_HEAD(&priv->event_nh); + mutex_init(&priv->port_mutex); mutex_init(&priv->bond_mutex); diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index de5699a4ddaa..8dbea814a2f5 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -47,6 +47,7 @@ #include #include #include +#include #include #include @@ -878,6 +879,7 @@ struct mlx4_priv { struct list_head dev_list; struct list_head ctx_list; spinlock_t ctx_lock; + struct atomic_notifier_head event_nh; int pci_dev_data; int removed; diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h index 72a3fea36702..efe3f97b874f 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h @@ -49,6 +49,7 @@ #include #include #include +#include #include #include @@ -433,6 +434,7 @@ struct mlx4_en_dev { struct ptp_clock *ptp_clock; struct ptp_clock_info ptp_clock_info; struct notifier_block netdev_nb; + struct notifier_block mlx_nb; }; diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 032d7f5bfef6..228da8ed7e75 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -34,6 +34,7 @@ #define MLX4_DRIVER_H #include +#include #include struct mlx4_dev; @@ -57,8 +58,6 @@ enum { struct mlx4_interface { void * (*add) (struct mlx4_dev *dev); void (*remove)(struct mlx4_dev *dev, void *context); - void (*event) (struct mlx4_dev *dev, void *context, - enum mlx4_dev_event event, void *param); void (*activate)(struct mlx4_dev *dev, void *context); struct list_head list; enum mlx4_protocol protocol; @@ -87,6 +86,11 @@ struct mlx4_port_map { int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p); +int mlx4_register_event_notifier(struct mlx4_dev *dev, + struct notifier_block *nb); +int mlx4_unregister_event_notifier(struct mlx4_dev *dev, + struct notifier_block *nb); + struct devlink_port *mlx4_get_devlink_port(struct mlx4_dev *dev, int port); #endif /* MLX4_DRIVER_H */ From patchwork Mon Aug 21 13:12:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359430 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA064EE4993 for ; Mon, 21 Aug 2023 13:12:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235257AbjHUNMw (ORCPT ); Mon, 21 Aug 2023 09:12:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235276AbjHUNMw (ORCPT ); Mon, 21 Aug 2023 09:12:52 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3318AF4; Mon, 21 Aug 2023 06:12:43 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id C679E2208F; Mon, 21 Aug 2023 13:12:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623561; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L8x9w82Z8aEgZifrI6JcH/JdzXmZ2Ssp/vWi4NElA70=; b=n1H8qcUJSRXtLTkA/2nB77XinOAD+k+L5AslkxzUzb96MHPA+YIdGmU/A0/1tY9Cc732mS K/3CwmaTuTbnSNpIIHl7JZkN9kkvaNXrSREZrGa9q++dNgD1gLhkkhriZmbKlQFWHyQT0w p+Wq3kkUonZUFTKS/4G0mpV2CJI6tn0= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 8570513421; Mon, 21 Aug 2023 13:12:41 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id MKJ3H8li42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:41 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v3 05/11] mlx4: Get rid of the mlx4_interface.activate callback Date: Mon, 21 Aug 2023 15:12:19 +0200 Message-Id: <20230821131225.11290-6-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The mlx4_interface.activate callback was introduced in commit 79857cd31fe7 ("net/mlx4: Postpone the registration of net_device"). It dealt with a situation when a netdev notifier received a NETDEV_REGISTER event for a new net_device created by mlx4_en but the same device was not yet visible to mlx4_get_protocol_dev(). The callback can be removed now that mlx4_get_protocol_dev() is gone. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/en_main.c | 37 +++++++++----------- drivers/net/ethernet/mellanox/mlx4/intf.c | 2 -- include/linux/mlx4/driver.h | 1 - 3 files changed, 16 insertions(+), 24 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index 31bf625b8158..39c9befeb638 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -259,26 +259,6 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) kfree(mdev); } -static void mlx4_en_activate(struct mlx4_dev *dev, void *ctx) -{ - int i; - struct mlx4_en_dev *mdev = ctx; - - /* Create a netdev for each port */ - mlx4_foreach_port(i, dev, MLX4_PORT_TYPE_ETH) { - mlx4_info(mdev, "Activating port:%d\n", i); - if (mlx4_en_init_netdev(mdev, i, &mdev->profile.prof[i])) - mdev->pndev[i] = NULL; - } - - /* register netdev notifier */ - mdev->netdev_nb.notifier_call = mlx4_en_netdev_event; - if (register_netdevice_notifier(&mdev->netdev_nb)) { - mdev->netdev_nb.notifier_call = NULL; - mlx4_err(mdev, "Failed to create netdev notifier\n"); - } -} - static void *mlx4_en_add(struct mlx4_dev *dev) { struct mlx4_en_dev *mdev; @@ -350,6 +330,22 @@ static void *mlx4_en_add(struct mlx4_dev *dev) err = mlx4_register_event_notifier(dev, &mdev->mlx_nb); WARN(err, "failed to register mlx4 event notifier (%d)", err); + /* Setup ports */ + + /* Create a netdev for each port */ + mlx4_foreach_port(i, dev, MLX4_PORT_TYPE_ETH) { + mlx4_info(mdev, "Activating port:%d\n", i); + if (mlx4_en_init_netdev(mdev, i, &mdev->profile.prof[i])) + mdev->pndev[i] = NULL; + } + + /* register netdev notifier */ + mdev->netdev_nb.notifier_call = mlx4_en_netdev_event; + if (register_netdevice_notifier(&mdev->netdev_nb)) { + mdev->netdev_nb.notifier_call = NULL; + mlx4_err(mdev, "Failed to create netdev notifier\n"); + } + return mdev; err_mr: @@ -371,7 +367,6 @@ static struct mlx4_interface mlx4_en_interface = { .add = mlx4_en_add, .remove = mlx4_en_remove, .protocol = MLX4_PROT_ETH, - .activate = mlx4_en_activate, }; static void mlx4_en_verify_params(void) diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index fecb63e69607..8cbc1bcdfe77 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -64,8 +64,6 @@ static void mlx4_add_device(struct mlx4_interface *intf, struct mlx4_priv *priv) spin_lock_irq(&priv->ctx_lock); list_add_tail(&dev_ctx->list, &priv->ctx_list); spin_unlock_irq(&priv->ctx_lock); - if (intf->activate) - intf->activate(&priv->dev, dev_ctx->context); } else kfree(dev_ctx); diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 228da8ed7e75..0f8c9ba4c574 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -58,7 +58,6 @@ enum { struct mlx4_interface { void * (*add) (struct mlx4_dev *dev); void (*remove)(struct mlx4_dev *dev, void *context); - void (*activate)(struct mlx4_dev *dev, void *context); struct list_head list; enum mlx4_protocol protocol; int flags; From patchwork Mon Aug 21 13:12:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48316EE4993 for ; Mon, 21 Aug 2023 13:12:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235297AbjHUNM5 (ORCPT ); Mon, 21 Aug 2023 09:12:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235282AbjHUNMw (ORCPT ); Mon, 21 Aug 2023 09:12:52 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62FF48F; Mon, 21 Aug 2023 06:12:43 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 11B8B206BF; Mon, 21 Aug 2023 13:12:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623562; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EF16vsCa6I0E6ur/tIvsNitlcraEXCA4gqMYY5ndwC4=; b=gw+BOQOOTzgY57mnqKx8/J/mW75JLGObYUeVwU5LqupgQo38+EkqHBP4dWp5rqpBG0Zf6a adAsJ8EPWAdkGxqoU+GUo6C+zIN7TxCfiAv921PsGixc6jr2r9JetleixIf/WWLu2TtUwJ 8pOSdImJ3SIJnk+XnRLiYhYrWnP/8RY= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id CBD20139BC; Mon, 21 Aug 2023 13:12:41 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id EBkAMcli42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:41 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v3 06/11] mlx4: Move the bond work to the core driver Date: Mon, 21 Aug 2023 15:12:20 +0200 Message-Id: <20230821131225.11290-7-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Function mlx4_en_queue_bond_work() is used in mlx4_en to start a bond reconfiguration. It gathers data about a new port map setting, takes a reference on the netdev that triggered the change and queues a work object on mlx4_en_priv.mdev.workqueue to perform the operation. The scheduled work is mlx4_en_bond_work() which calls mlx4_bond()/mlx4_unbond() and consequently mlx4_do_bond(). At the same time, function mlx4_change_port_types() in mlx4_core might be invoked to change the port type configuration. As part of its logic, it re-registers the whole device by calling mlx4_unregister_device(), followed by mlx4_register_device(). The two operations can result in concurrent access to the data about currently active interfaces on the device. Functions mlx4_register_device() and mlx4_unregister_device() lock the intf_mutex to gain exclusive access to this data. The current implementation of mlx4_do_bond() doesn't do that which could result in an unexpected behavior. An updated version of mlx4_do_bond() for use with an auxiliary bus goes and locks the intf_mutex when accessing a new auxiliary device array. However, doing so can then result in the following deadlock: * A two-port mlx4 device is configured as an Ethernet bond. * One of the ports is changed from eth to ib, for instance, by writing into a mlx4_port sysfs attribute file. * mlx4_change_port_types() is called to update port types. It invokes mlx4_unregister_device() to unregister the device which locks the intf_mutex and starts removing all associated interfaces. * Function mlx4_en_remove() gets invoked and starts destroying its first netdev. This triggers mlx4_en_netdev_event() which recognizes that the configured bond is broken. It runs mlx4_en_queue_bond_work() which takes a reference on the netdev. Removing the netdev now cannot proceed until the work is completed. * Work function mlx4_en_bond_work() gets scheduled. It calls mlx4_unbond() -> mlx4_do_bond(). The latter function tries to lock the intf_mutex but that is not possible because it is held already by mlx4_unregister_device(). This particular case could be possibly solved by unregistering the mlx4_en_netdev_event() notifier in mlx4_en_remove() earlier, but it seems better to decouple mlx4_en more and break this reference order. Avoid then this scenario by recognizing that the bond reconfiguration operates only on a mlx4_dev. The logic to queue and execute the bond work can be moved into the mlx4_core driver. Only a reference on the respective mlx4_dev object is needed to be taken during the work's lifetime. This removes a call from mlx4_en that can directly result in needing to lock the intf_mutex, it remains a privilege of the core driver. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- .../net/ethernet/mellanox/mlx4/en_netdev.c | 62 +----------------- drivers/net/ethernet/mellanox/mlx4/main.c | 65 +++++++++++++++++-- drivers/net/ethernet/mellanox/mlx4/mlx4.h | 5 ++ include/linux/mlx4/device.h | 13 ++++ include/linux/mlx4/driver.h | 19 ------ 5 files changed, 77 insertions(+), 87 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c index 7066c426b95c..33bbcced8105 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c @@ -2894,63 +2894,6 @@ static const struct xdp_metadata_ops mlx4_xdp_metadata_ops = { .xmo_rx_hash = mlx4_en_xdp_rx_hash, }; -struct mlx4_en_bond { - struct work_struct work; - struct mlx4_en_priv *priv; - int is_bonded; - struct mlx4_port_map port_map; -}; - -static void mlx4_en_bond_work(struct work_struct *work) -{ - struct mlx4_en_bond *bond = container_of(work, - struct mlx4_en_bond, - work); - int err = 0; - struct mlx4_dev *dev = bond->priv->mdev->dev; - - if (bond->is_bonded) { - if (!mlx4_is_bonded(dev)) { - err = mlx4_bond(dev); - if (err) - en_err(bond->priv, "Fail to bond device\n"); - } - if (!err) { - err = mlx4_port_map_set(dev, &bond->port_map); - if (err) - en_err(bond->priv, "Fail to set port map [%d][%d]: %d\n", - bond->port_map.port1, - bond->port_map.port2, - err); - } - } else if (mlx4_is_bonded(dev)) { - err = mlx4_unbond(dev); - if (err) - en_err(bond->priv, "Fail to unbond device\n"); - } - dev_put(bond->priv->dev); - kfree(bond); -} - -static int mlx4_en_queue_bond_work(struct mlx4_en_priv *priv, int is_bonded, - u8 v2p_p1, u8 v2p_p2) -{ - struct mlx4_en_bond *bond; - - bond = kzalloc(sizeof(*bond), GFP_ATOMIC); - if (!bond) - return -ENOMEM; - - INIT_WORK(&bond->work, mlx4_en_bond_work); - bond->priv = priv; - bond->is_bonded = is_bonded; - bond->port_map.port1 = v2p_p1; - bond->port_map.port2 = v2p_p2; - dev_hold(priv->dev); - queue_work(priv->mdev->workqueue, &bond->work); - return 0; -} - int mlx4_en_netdev_event(struct notifier_block *this, unsigned long event, void *ptr) { @@ -2960,7 +2903,6 @@ int mlx4_en_netdev_event(struct notifier_block *this, struct mlx4_dev *dev; int i, num_eth_ports = 0; bool do_bond = true; - struct mlx4_en_priv *priv; u8 v2p_port1 = 0; u8 v2p_port2 = 0; @@ -2995,7 +2937,6 @@ int mlx4_en_netdev_event(struct notifier_block *this, if ((do_bond && (event != NETDEV_BONDING_INFO)) || !port) return NOTIFY_DONE; - priv = netdev_priv(ndev); if (do_bond) { struct netdev_notifier_bonding_info *notifier_info = ptr; struct netdev_bonding_info *bonding_info = @@ -3062,8 +3003,7 @@ int mlx4_en_netdev_event(struct notifier_block *this, } } - mlx4_en_queue_bond_work(priv, do_bond, - v2p_port1, v2p_port2); + mlx4_queue_bond_work(dev, do_bond, v2p_port1, v2p_port2); return NOTIFY_DONE; } diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index 5f3ba8385e23..0ed490b99163 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -1441,7 +1441,7 @@ static int mlx4_mf_unbond(struct mlx4_dev *dev) return ret; } -int mlx4_bond(struct mlx4_dev *dev) +static int mlx4_bond(struct mlx4_dev *dev) { int ret = 0; struct mlx4_priv *priv = mlx4_priv(dev); @@ -1467,9 +1467,8 @@ int mlx4_bond(struct mlx4_dev *dev) return ret; } -EXPORT_SYMBOL_GPL(mlx4_bond); -int mlx4_unbond(struct mlx4_dev *dev) +static int mlx4_unbond(struct mlx4_dev *dev) { int ret = 0; struct mlx4_priv *priv = mlx4_priv(dev); @@ -1496,10 +1495,8 @@ int mlx4_unbond(struct mlx4_dev *dev) return ret; } -EXPORT_SYMBOL_GPL(mlx4_unbond); - -int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p) +static int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p) { u8 port1 = v2p->port1; u8 port2 = v2p->port2; @@ -1541,7 +1538,61 @@ int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p) mutex_unlock(&priv->bond_mutex); return err; } -EXPORT_SYMBOL_GPL(mlx4_port_map_set); + +struct mlx4_bond { + struct work_struct work; + struct mlx4_dev *dev; + int is_bonded; + struct mlx4_port_map port_map; +}; + +static void mlx4_bond_work(struct work_struct *work) +{ + struct mlx4_bond *bond = container_of(work, struct mlx4_bond, work); + int err = 0; + + if (bond->is_bonded) { + if (!mlx4_is_bonded(bond->dev)) { + err = mlx4_bond(bond->dev); + if (err) + mlx4_err(bond->dev, "Fail to bond device\n"); + } + if (!err) { + err = mlx4_port_map_set(bond->dev, &bond->port_map); + if (err) + mlx4_err(bond->dev, + "Fail to set port map [%d][%d]: %d\n", + bond->port_map.port1, + bond->port_map.port2, err); + } + } else if (mlx4_is_bonded(bond->dev)) { + err = mlx4_unbond(bond->dev); + if (err) + mlx4_err(bond->dev, "Fail to unbond device\n"); + } + put_device(&bond->dev->persist->pdev->dev); + kfree(bond); +} + +int mlx4_queue_bond_work(struct mlx4_dev *dev, int is_bonded, u8 v2p_p1, + u8 v2p_p2) +{ + struct mlx4_bond *bond; + + bond = kzalloc(sizeof(*bond), GFP_ATOMIC); + if (!bond) + return -ENOMEM; + + INIT_WORK(&bond->work, mlx4_bond_work); + get_device(&dev->persist->pdev->dev); + bond->dev = dev; + bond->is_bonded = is_bonded; + bond->port_map.port1 = v2p_p1; + bond->port_map.port2 = v2p_p2; + queue_work(mlx4_wq, &bond->work); + return 0; +} +EXPORT_SYMBOL(mlx4_queue_bond_work); static int mlx4_load_fw(struct mlx4_dev *dev) { diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index 8dbea814a2f5..73b4fdfd63f4 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -863,6 +863,11 @@ struct mlx4_steer { struct list_head steer_entries[MLX4_NUM_STEERS]; }; +struct mlx4_port_map { + u8 port1; + u8 port2; +}; + enum { MLX4_PCI_DEV_IS_VF = 1 << 0, MLX4_PCI_DEV_FORCE_SENSE_PORT = 1 << 1, diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index 6646634a0b9d..049d8a4b044d 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -1087,6 +1087,19 @@ static inline void *mlx4_buf_offset(struct mlx4_buf *buf, int offset) (offset & (PAGE_SIZE - 1)); } +static inline int mlx4_is_bonded(struct mlx4_dev *dev) +{ + return !!(dev->flags & MLX4_FLAG_BONDED); +} + +static inline int mlx4_is_mf_bonded(struct mlx4_dev *dev) +{ + return (mlx4_is_bonded(dev) && mlx4_is_mfunc(dev)); +} + +int mlx4_queue_bond_work(struct mlx4_dev *dev, int is_bonded, u8 v2p_p1, + u8 v2p_p2); + int mlx4_pd_alloc(struct mlx4_dev *dev, u32 *pdn); void mlx4_pd_free(struct mlx4_dev *dev, u32 pdn); int mlx4_xrcd_alloc(struct mlx4_dev *dev, u32 *xrcdn); diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 0f8c9ba4c574..781d5a0c2faa 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -66,25 +66,6 @@ struct mlx4_interface { int mlx4_register_interface(struct mlx4_interface *intf); void mlx4_unregister_interface(struct mlx4_interface *intf); -int mlx4_bond(struct mlx4_dev *dev); -int mlx4_unbond(struct mlx4_dev *dev); -static inline int mlx4_is_bonded(struct mlx4_dev *dev) -{ - return !!(dev->flags & MLX4_FLAG_BONDED); -} - -static inline int mlx4_is_mf_bonded(struct mlx4_dev *dev) -{ - return (mlx4_is_bonded(dev) && mlx4_is_mfunc(dev)); -} - -struct mlx4_port_map { - u8 port1; - u8 port2; -}; - -int mlx4_port_map_set(struct mlx4_dev *dev, struct mlx4_port_map *v2p); - int mlx4_register_event_notifier(struct mlx4_dev *dev, struct notifier_block *nb); int mlx4_unregister_event_notifier(struct mlx4_dev *dev, From patchwork Mon Aug 21 13:12:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4ED8EE49AC for ; Mon, 21 Aug 2023 13:12:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235306AbjHUNM6 (ORCPT ); Mon, 21 Aug 2023 09:12:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235295AbjHUNM5 (ORCPT ); Mon, 21 Aug 2023 09:12:57 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7AFD6EC; Mon, 21 Aug 2023 06:12:49 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 541BE206C0; Mon, 21 Aug 2023 13:12:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623562; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BrDZ69H2NWA/XUC/sTqplN7GhGldnmZpsSLx8bYevfo=; b=g01CvYRTAQaUQKyygK+ZdRr1dpgLaJdZk8rh7HPL2JcUgjSzEUPe+bKzngBGcLcLgzyhMj QeJyipM9N4RrRF9t7wIr8CIaXwEkFEWxvdH61dvNXCWI/aTPWXAQALozcX9kGMZQPNHvSS 5ztLP1pxuYkpUeduMSAz4/uuS/lQqN0= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 165C613421; Mon, 21 Aug 2023 13:12:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id cCCiBMpi42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:42 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v3 07/11] mlx4: Avoid resetting MLX4_INTFF_BONDING per driver Date: Mon, 21 Aug 2023 15:12:21 +0200 Message-Id: <20230821131225.11290-8-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The mlx4_core driver has a logic that allows a sub-driver to set the MLX4_INTFF_BONDING flag which then causes that function mlx4_do_bond() asks the sub-driver to fully re-probe a device when its bonding configuration changes. Performing this operation is disallowed in mlx4_register_interface() when it is detected that any mlx4 device is multifunction (SRIOV). The code then resets MLX4_INTFF_BONDING in the driver flags. Move this check directly into mlx4_do_bond(). It provides a better separation as mlx4_core no longer directly modifies the sub-driver flags and it will allow to get rid of explicitly keeping track of all mlx4 devices by the intf.c code when it is switched to an auxiliary bus. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/intf.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 8cbc1bcdfe77..d578f754d882 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -96,11 +96,6 @@ int mlx4_register_interface(struct mlx4_interface *intf) list_add_tail(&intf->list, &intf_list); list_for_each_entry(priv, &dev_list, dev_list) { - if (mlx4_is_mfunc(&priv->dev) && (intf->flags & MLX4_INTFF_BONDING)) { - mlx4_dbg(&priv->dev, - "SRIOV, disabling HA mode for intf proto %d\n", intf->protocol); - intf->flags &= ~MLX4_INTFF_BONDING; - } mlx4_add_device(intf, priv); } @@ -155,10 +150,18 @@ int mlx4_do_bond(struct mlx4_dev *dev, bool enable) spin_lock_irqsave(&priv->ctx_lock, flags); list_for_each_entry_safe(dev_ctx, temp_dev_ctx, &priv->ctx_list, list) { - if (dev_ctx->intf->flags & MLX4_INTFF_BONDING) { - list_add_tail(&dev_ctx->bond_list, &bond_list); - list_del(&dev_ctx->list); + if (!(dev_ctx->intf->flags & MLX4_INTFF_BONDING)) + continue; + + if (mlx4_is_mfunc(dev)) { + mlx4_dbg(dev, + "SRIOV, disabled HA mode for intf proto %d\n", + dev_ctx->intf->protocol); + continue; } + + list_add_tail(&dev_ctx->bond_list, &bond_list); + list_del(&dev_ctx->list); } spin_unlock_irqrestore(&priv->ctx_lock, flags); From patchwork Mon Aug 21 13:12:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75A03EE49B2 for ; Mon, 21 Aug 2023 13:12:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235281AbjHUNM7 (ORCPT ); Mon, 21 Aug 2023 09:12:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235300AbjHUNM5 (ORCPT ); Mon, 21 Aug 2023 09:12:57 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2001:67c:2178:6::1d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7AF51EB; Mon, 21 Aug 2023 06:12:49 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 9EFAB206C1; Mon, 21 Aug 2023 13:12:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623562; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xT/61KJiTZugj2y9kn0d2eSXJsIjB9+fo/ykA8JwKz4=; b=KT9fGI6HB3s+2heKiypitb6XVFw2fp93Xr2CovwMYwyMEspM8IckoYLeM1v1p5AjGYHKyU WwzOrbMj1E+Pf7Nk1JxVBCrOTOg6u+cIjlK9JJLoXwzQroHzCnXOXbrKQ1g6pYsCRtd8Dk KUDyW3130bZSq63rojuz7848vh2Qm7o= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 5BAC813421; Mon, 21 Aug 2023 13:12:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id iNpGFcpi42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:42 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v3 08/11] mlx4: Register mlx4 devices to an auxiliary virtual bus Date: Mon, 21 Aug 2023 15:12:22 +0200 Message-Id: <20230821131225.11290-9-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Add an auxiliary virtual bus to model the mlx4 driver structure. The code is added along the current custom device management logic. Subsequent patches switch mlx4_en and mlx4_ib to the auxiliary bus and the old interface is then removed. Structure mlx4_priv gains a new adev dynamic array to keep track of its auxiliary devices. Access to the array is protected by the global mlx4_intf mutex. Functions mlx4_register_device() and mlx4_unregister_device() are updated to expose auxiliary devices on the bus in order to load mlx4_en and/or mlx4_ib. Functions mlx4_register_auxiliary_driver() and mlx4_unregister_auxiliary_driver() are added to substitute mlx4_register_interface() and mlx4_unregister_interface(), respectively. Function mlx4_do_bond() is adjusted to walk over the adev array and re-adds a specific auxiliary device if its driver sets the MLX4_INTFF_BONDING flag. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/Kconfig | 1 + drivers/net/ethernet/mellanox/mlx4/intf.c | 230 ++++++++++++++++++++- drivers/net/ethernet/mellanox/mlx4/main.c | 17 +- drivers/net/ethernet/mellanox/mlx4/mlx4.h | 6 + include/linux/mlx4/device.h | 7 + include/linux/mlx4/driver.h | 11 + 6 files changed, 268 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/Kconfig b/drivers/net/ethernet/mellanox/mlx4/Kconfig index 1b4b1f642317..825e05fb8607 100644 --- a/drivers/net/ethernet/mellanox/mlx4/Kconfig +++ b/drivers/net/ethernet/mellanox/mlx4/Kconfig @@ -27,6 +27,7 @@ config MLX4_EN_DCB config MLX4_CORE tristate depends on PCI + select AUXILIARY_BUS select NET_DEVLINK default n diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index d578f754d882..79b1eaa62ac2 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -48,6 +48,89 @@ struct mlx4_device_context { static LIST_HEAD(intf_list); static LIST_HEAD(dev_list); static DEFINE_MUTEX(intf_mutex); +static DEFINE_IDA(mlx4_adev_ida); + +static const struct mlx4_adev_device { + const char *suffix; + bool (*is_supported)(struct mlx4_dev *dev); +} mlx4_adev_devices[1] = {}; + +int mlx4_adev_init(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + priv->adev_idx = ida_alloc(&mlx4_adev_ida, GFP_KERNEL); + if (priv->adev_idx < 0) + return priv->adev_idx; + + priv->adev = kcalloc(ARRAY_SIZE(mlx4_adev_devices), + sizeof(struct mlx4_adev *), GFP_KERNEL); + if (!priv->adev) { + ida_free(&mlx4_adev_ida, priv->adev_idx); + return -ENOMEM; + } + + return 0; +} + +void mlx4_adev_cleanup(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + + kfree(priv->adev); + ida_free(&mlx4_adev_ida, priv->adev_idx); +} + +static void adev_release(struct device *dev) +{ + struct mlx4_adev *mlx4_adev = + container_of(dev, struct mlx4_adev, adev.dev); + struct mlx4_priv *priv = mlx4_priv(mlx4_adev->mdev); + int idx = mlx4_adev->idx; + + kfree(mlx4_adev); + priv->adev[idx] = NULL; +} + +static struct mlx4_adev *add_adev(struct mlx4_dev *dev, int idx) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + const char *suffix = mlx4_adev_devices[idx].suffix; + struct auxiliary_device *adev; + struct mlx4_adev *madev; + int ret; + + madev = kzalloc(sizeof(*madev), GFP_KERNEL); + if (!madev) + return ERR_PTR(-ENOMEM); + + adev = &madev->adev; + adev->id = priv->adev_idx; + adev->name = suffix; + adev->dev.parent = &dev->persist->pdev->dev; + adev->dev.release = adev_release; + madev->mdev = dev; + madev->idx = idx; + + ret = auxiliary_device_init(adev); + if (ret) { + kfree(madev); + return ERR_PTR(ret); + } + + ret = auxiliary_device_add(adev); + if (ret) { + auxiliary_device_uninit(adev); + return ERR_PTR(ret); + } + return madev; +} + +static void del_adev(struct auxiliary_device *adev) +{ + auxiliary_device_delete(adev); + auxiliary_device_uninit(adev); +} static void mlx4_add_device(struct mlx4_interface *intf, struct mlx4_priv *priv) { @@ -120,12 +203,24 @@ void mlx4_unregister_interface(struct mlx4_interface *intf) } EXPORT_SYMBOL_GPL(mlx4_unregister_interface); +int mlx4_register_auxiliary_driver(struct mlx4_adrv *madrv) +{ + return auxiliary_driver_register(&madrv->adrv); +} +EXPORT_SYMBOL_GPL(mlx4_register_auxiliary_driver); + +void mlx4_unregister_auxiliary_driver(struct mlx4_adrv *madrv) +{ + auxiliary_driver_unregister(&madrv->adrv); +} +EXPORT_SYMBOL_GPL(mlx4_unregister_auxiliary_driver); + int mlx4_do_bond(struct mlx4_dev *dev, bool enable) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_device_context *dev_ctx = NULL, *temp_dev_ctx; unsigned long flags; - int ret; + int i, ret; LIST_HEAD(bond_list); if (!(dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_PORT_REMAP)) @@ -177,6 +272,57 @@ int mlx4_do_bond(struct mlx4_dev *dev, bool enable) dev_ctx->intf->protocol, enable ? "enabled" : "disabled"); } + + mutex_lock(&intf_mutex); + + for (i = 0; i < ARRAY_SIZE(mlx4_adev_devices); i++) { + struct mlx4_adev *madev = priv->adev[i]; + struct mlx4_adrv *madrv; + enum mlx4_protocol protocol; + + if (!madev) + continue; + + device_lock(&madev->adev.dev); + if (!madev->adev.dev.driver) { + device_unlock(&madev->adev.dev); + continue; + } + + madrv = container_of(madev->adev.dev.driver, struct mlx4_adrv, + adrv.driver); + if (!(madrv->flags & MLX4_INTFF_BONDING)) { + device_unlock(&madev->adev.dev); + continue; + } + + if (mlx4_is_mfunc(dev)) { + mlx4_dbg(dev, + "SRIOV, disabled HA mode for intf proto %d\n", + madrv->protocol); + device_unlock(&madev->adev.dev); + continue; + } + + protocol = madrv->protocol; + device_unlock(&madev->adev.dev); + + del_adev(&madev->adev); + priv->adev[i] = add_adev(dev, i); + if (IS_ERR(priv->adev[i])) { + mlx4_warn(dev, "Device[%d] (%s) failed to load\n", i, + mlx4_adev_devices[i].suffix); + priv->adev[i] = NULL; + continue; + } + + mlx4_dbg(dev, + "Interface for protocol %d restarted with bonded mode %s\n", + protocol, enable ? "enabled" : "disabled"); + } + + mutex_unlock(&intf_mutex); + return 0; } @@ -206,10 +352,80 @@ int mlx4_unregister_event_notifier(struct mlx4_dev *dev, } EXPORT_SYMBOL(mlx4_unregister_event_notifier); +static int add_drivers(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + int i, ret = 0; + + for (i = 0; i < ARRAY_SIZE(mlx4_adev_devices); i++) { + bool is_supported = false; + + if (priv->adev[i]) + continue; + + if (mlx4_adev_devices[i].is_supported) + is_supported = mlx4_adev_devices[i].is_supported(dev); + + if (!is_supported) + continue; + + priv->adev[i] = add_adev(dev, i); + if (IS_ERR(priv->adev[i])) { + mlx4_warn(dev, "Device[%d] (%s) failed to load\n", i, + mlx4_adev_devices[i].suffix); + /* We continue to rescan drivers and leave to the caller + * to make decision if to release everything or + * continue. */ + ret = PTR_ERR(priv->adev[i]); + priv->adev[i] = NULL; + } + } + return ret; +} + +static void delete_drivers(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + bool delete_all; + int i; + + delete_all = !(dev->persist->interface_state & MLX4_INTERFACE_STATE_UP); + + for (i = ARRAY_SIZE(mlx4_adev_devices) - 1; i >= 0; i--) { + bool is_supported = false; + + if (!priv->adev[i]) + continue; + + if (mlx4_adev_devices[i].is_supported && !delete_all) + is_supported = mlx4_adev_devices[i].is_supported(dev); + + if (is_supported) + continue; + + del_adev(&priv->adev[i]->adev); + priv->adev[i] = NULL; + } +} + +/* This function is used after mlx4_dev is reconfigured. + */ +static int rescan_drivers_locked(struct mlx4_dev *dev) +{ + lockdep_assert_held(&intf_mutex); + + delete_drivers(dev); + if (!(dev->persist->interface_state & MLX4_INTERFACE_STATE_UP)) + return 0; + + return add_drivers(dev); +} + int mlx4_register_device(struct mlx4_dev *dev) { struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_interface *intf; + int ret; mutex_lock(&intf_mutex); @@ -218,10 +434,18 @@ int mlx4_register_device(struct mlx4_dev *dev) list_for_each_entry(intf, &intf_list, list) mlx4_add_device(intf, priv); + ret = rescan_drivers_locked(dev); + mutex_unlock(&intf_mutex); + + if (ret) { + mlx4_unregister_device(dev); + return ret; + } + mlx4_start_catas_poll(dev); - return 0; + return ret; } void mlx4_unregister_device(struct mlx4_dev *dev) @@ -253,6 +477,8 @@ void mlx4_unregister_device(struct mlx4_dev *dev) list_del(&priv->dev_list); dev->persist->interface_state &= ~MLX4_INTERFACE_STATE_UP; + rescan_drivers_locked(dev); + mutex_unlock(&intf_mutex); } diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index 0ed490b99163..c4ec7377aa71 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -3429,6 +3429,10 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, INIT_LIST_HEAD(&priv->ctx_list); spin_lock_init(&priv->ctx_lock); + err = mlx4_adev_init(dev); + if (err) + return err; + ATOMIC_INIT_NOTIFIER_HEAD(&priv->event_nh); mutex_init(&priv->port_mutex); @@ -3455,10 +3459,11 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, err = mlx4_get_ownership(dev); if (err) { if (err < 0) - return err; + goto err_adev; else { mlx4_warn(dev, "Multiple PFs not yet supported - Skipping PF\n"); - return -EINVAL; + err = -EINVAL; + goto err_adev; } } @@ -3806,6 +3811,9 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, mlx4_free_ownership(dev); kfree(dev_cap); + +err_adev: + mlx4_adev_cleanup(dev); return err; } @@ -4186,6 +4194,8 @@ static void mlx4_unload_one(struct pci_dev *pdev) mlx4_slave_destroy_special_qp_cap(dev); kfree(dev->dev_vfs); + mlx4_adev_cleanup(dev); + mlx4_clean_dev(dev); priv->pci_dev_data = pci_dev_data; priv->removed = 1; @@ -4573,6 +4583,9 @@ static int __init mlx4_init(void) { int ret; + WARN_ONCE(strcmp(MLX4_ADEV_NAME, KBUILD_MODNAME), + "mlx4_core name not in sync with kernel module name"); + if (mlx4_verify_params()) return -EINVAL; diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index 73b4fdfd63f4..36de79c8d053 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -47,6 +47,7 @@ #include #include #include +#include #include #include @@ -884,6 +885,8 @@ struct mlx4_priv { struct list_head dev_list; struct list_head ctx_list; spinlock_t ctx_lock; + struct mlx4_adev **adev; + int adev_idx; struct atomic_notifier_head event_nh; int pci_dev_data; @@ -1052,6 +1055,9 @@ void mlx4_catas_end(struct mlx4_dev *dev); int mlx4_crdump_init(struct mlx4_dev *dev); void mlx4_crdump_end(struct mlx4_dev *dev); int mlx4_restart_one(struct pci_dev *pdev); + +int mlx4_adev_init(struct mlx4_dev *dev); +void mlx4_adev_cleanup(struct mlx4_dev *dev); int mlx4_register_device(struct mlx4_dev *dev); void mlx4_unregister_device(struct mlx4_dev *dev); void mlx4_dispatch_event(struct mlx4_dev *dev, enum mlx4_dev_event type, diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index 049d8a4b044d..27f42f713c89 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -33,6 +33,7 @@ #ifndef MLX4_DEVICE_H #define MLX4_DEVICE_H +#include #include #include #include @@ -889,6 +890,12 @@ struct mlx4_dev { u8 uar_page_shift; }; +struct mlx4_adev { + struct auxiliary_device adev; + struct mlx4_dev *mdev; + int idx; +}; + struct mlx4_clock_params { u64 offset; u8 bar; diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 781d5a0c2faa..9cf157d381c6 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -34,9 +34,12 @@ #define MLX4_DRIVER_H #include +#include #include #include +#define MLX4_ADEV_NAME "mlx4_core" + struct mlx4_dev; #define MLX4_MAC_MASK 0xffffffffffffULL @@ -63,8 +66,16 @@ struct mlx4_interface { int flags; }; +struct mlx4_adrv { + struct auxiliary_driver adrv; + enum mlx4_protocol protocol; + int flags; +}; + int mlx4_register_interface(struct mlx4_interface *intf); void mlx4_unregister_interface(struct mlx4_interface *intf); +int mlx4_register_auxiliary_driver(struct mlx4_adrv *madrv); +void mlx4_unregister_auxiliary_driver(struct mlx4_adrv *madrv); int mlx4_register_event_notifier(struct mlx4_dev *dev, struct notifier_block *nb); From patchwork Mon Aug 21 13:12:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359438 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BF79EE49B1 for ; Mon, 21 Aug 2023 13:13:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235318AbjHUNM7 (ORCPT ); Mon, 21 Aug 2023 09:12:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235301AbjHUNM5 (ORCPT ); Mon, 21 Aug 2023 09:12:57 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 228F4100; Mon, 21 Aug 2023 06:12:50 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E4710206C2; Mon, 21 Aug 2023 13:12:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623562; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U+/IaM7P+6aeGaih/C39diNuw+MT+GSi2NQS+0njdzo=; b=kmd0ViEqt1asfAGfNrIqERcnQpv9Do1tZvfF3zVRIN3Ei5UF9XBVXU62jvEoiTSmG0MaXK 6EEBTiA4doiS3723448lPW+LSOOvXlAYUx3YgQRgoMkJglPIIN0GtKp/Q2T89ZvXTohzsz ozbLXkQyjnKfhMFkpPRxcuxIunxJvd4= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id A2939139BC; Mon, 21 Aug 2023 13:12:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id sM7NJspi42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:42 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v3 09/11] mlx4: Connect the ethernet part to the auxiliary bus Date: Mon, 21 Aug 2023 15:12:23 +0200 Message-Id: <20230821131225.11290-10-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Use the auxiliary bus to perform device management of the ethernet part of the mlx4 driver. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/en_main.c | 65 ++++++++++++++------ drivers/net/ethernet/mellanox/mlx4/intf.c | 13 +++- 2 files changed, 58 insertions(+), 20 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/en_main.c b/drivers/net/ethernet/mellanox/mlx4/en_main.c index 39c9befeb638..d8f4d00ad26b 100644 --- a/drivers/net/ethernet/mellanox/mlx4/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx4/en_main.c @@ -234,9 +234,11 @@ static int mlx4_en_event(struct notifier_block *this, unsigned long event, return NOTIFY_DONE; } -static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) +static void mlx4_en_remove(struct auxiliary_device *adev) { - struct mlx4_en_dev *mdev = endev_ptr; + struct mlx4_adev *madev = container_of(adev, struct mlx4_adev, adev); + struct mlx4_dev *dev = madev->mdev; + struct mlx4_en_dev *mdev = auxiliary_get_drvdata(adev); int i; mlx4_unregister_event_notifier(dev, &mdev->mlx_nb); @@ -259,27 +261,36 @@ static void mlx4_en_remove(struct mlx4_dev *dev, void *endev_ptr) kfree(mdev); } -static void *mlx4_en_add(struct mlx4_dev *dev) +static int mlx4_en_probe(struct auxiliary_device *adev, + const struct auxiliary_device_id *id) { + struct mlx4_adev *madev = container_of(adev, struct mlx4_adev, adev); + struct mlx4_dev *dev = madev->mdev; struct mlx4_en_dev *mdev; int err, i; printk_once(KERN_INFO "%s", mlx4_en_version); mdev = kzalloc(sizeof(*mdev), GFP_KERNEL); - if (!mdev) + if (!mdev) { + err = -ENOMEM; goto err_free_res; + } - if (mlx4_pd_alloc(dev, &mdev->priv_pdn)) + err = mlx4_pd_alloc(dev, &mdev->priv_pdn); + if (err) goto err_free_dev; - if (mlx4_uar_alloc(dev, &mdev->priv_uar)) + err = mlx4_uar_alloc(dev, &mdev->priv_uar); + if (err) goto err_pd; mdev->uar_map = ioremap((phys_addr_t) mdev->priv_uar.pfn << PAGE_SHIFT, PAGE_SIZE); - if (!mdev->uar_map) + if (!mdev->uar_map) { + err = -ENOMEM; goto err_uar; + } spin_lock_init(&mdev->uar_lock); mdev->dev = dev; @@ -291,13 +302,15 @@ static void *mlx4_en_add(struct mlx4_dev *dev) if (!mdev->LSO_support) mlx4_warn(mdev, "LSO not supported, please upgrade to later FW version to enable LSO\n"); - if (mlx4_mr_alloc(mdev->dev, mdev->priv_pdn, 0, ~0ull, - MLX4_PERM_LOCAL_WRITE | MLX4_PERM_LOCAL_READ, - 0, 0, &mdev->mr)) { + err = mlx4_mr_alloc(mdev->dev, mdev->priv_pdn, 0, ~0ull, + MLX4_PERM_LOCAL_WRITE | MLX4_PERM_LOCAL_READ, 0, 0, + &mdev->mr); + if (err) { mlx4_err(mdev, "Failed allocating memory region\n"); goto err_map; } - if (mlx4_mr_enable(mdev->dev, &mdev->mr)) { + err = mlx4_mr_enable(mdev->dev, &mdev->mr); + if (err) { mlx4_err(mdev, "Failed enabling memory region\n"); goto err_mr; } @@ -317,8 +330,10 @@ static void *mlx4_en_add(struct mlx4_dev *dev) * Note: we cannot use the shared workqueue because of deadlocks caused * by the rtnl lock */ mdev->workqueue = create_singlethread_workqueue("mlx4_en"); - if (!mdev->workqueue) + if (!mdev->workqueue) { + err = -ENOMEM; goto err_mr; + } /* At this stage all non-port specific tasks are complete: * mark the card state as up */ @@ -346,7 +361,8 @@ static void *mlx4_en_add(struct mlx4_dev *dev) mlx4_err(mdev, "Failed to create netdev notifier\n"); } - return mdev; + auxiliary_set_drvdata(adev, mdev); + return 0; err_mr: (void) mlx4_mr_free(dev, &mdev->mr); @@ -360,12 +376,23 @@ static void *mlx4_en_add(struct mlx4_dev *dev) err_free_dev: kfree(mdev); err_free_res: - return NULL; + return err; } -static struct mlx4_interface mlx4_en_interface = { - .add = mlx4_en_add, - .remove = mlx4_en_remove, +static const struct auxiliary_device_id mlx4_en_id_table[] = { + { .name = MLX4_ADEV_NAME ".eth" }, + {}, +}; + +MODULE_DEVICE_TABLE(auxiliary, mlx4_en_id_table); + +static struct mlx4_adrv mlx4_en_adrv = { + .adrv = { + .name = "eth", + .probe = mlx4_en_probe, + .remove = mlx4_en_remove, + .id_table = mlx4_en_id_table, + }, .protocol = MLX4_PROT_ETH, }; @@ -395,12 +422,12 @@ static int __init mlx4_en_init(void) mlx4_en_verify_params(); mlx4_en_init_ptys2ethtool_map(); - return mlx4_register_interface(&mlx4_en_interface); + return mlx4_register_auxiliary_driver(&mlx4_en_adrv); } static void __exit mlx4_en_cleanup(void) { - mlx4_unregister_interface(&mlx4_en_interface); + mlx4_unregister_auxiliary_driver(&mlx4_en_adrv); } module_init(mlx4_en_init); diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 79b1eaa62ac2..95ff0ea435e2 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -50,10 +50,21 @@ static LIST_HEAD(dev_list); static DEFINE_MUTEX(intf_mutex); static DEFINE_IDA(mlx4_adev_ida); +static bool is_eth_supported(struct mlx4_dev *dev) +{ + for (int port = 1; port <= dev->caps.num_ports; port++) + if (dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH) + return true; + + return false; +} + static const struct mlx4_adev_device { const char *suffix; bool (*is_supported)(struct mlx4_dev *dev); -} mlx4_adev_devices[1] = {}; +} mlx4_adev_devices[] = { + { "eth", is_eth_supported }, +}; int mlx4_adev_init(struct mlx4_dev *dev) { From patchwork Mon Aug 21 13:12:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359434 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11431C3DA66 for ; Mon, 21 Aug 2023 13:12:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235303AbjHUNM6 (ORCPT ); Mon, 21 Aug 2023 09:12:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235283AbjHUNMx (ORCPT ); Mon, 21 Aug 2023 09:12:53 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [IPv6:2001:67c:2178:6::1c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78D9EF7; Mon, 21 Aug 2023 06:12:44 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 3A12D2217D; Mon, 21 Aug 2023 13:12:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623563; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ggdVss3a7x30oJhF7BsemZGfEN+xpQYOk9kDIS7xe8M=; b=nXJVssviQF1hyzNEG3jmWwJ63px0WYLkRLRliSPs9aYNV/9fYlz8bFa3m9OVbqREz0elJv gTL4GgBUNmCBCHEzpqKHdAazGbUUCrfpFDOo/sPUfOu/IKjKA5MSN+or9IPKUQmWz+S/hU NVNPvlII1FnEeGj2BOGsufFcA09ynP8= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id ED92F13421; Mon, 21 Aug 2023 13:12:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 8AIaOcpi42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:42 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v3 10/11] mlx4: Connect the infiniband part to the auxiliary bus Date: Mon, 21 Aug 2023 15:12:24 +0200 Message-Id: <20230821131225.11290-11-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Use the auxiliary bus to perform device management of the infiniband part of the mlx4 driver. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/infiniband/hw/mlx4/main.c | 77 ++++++++++++++++------- drivers/net/ethernet/mellanox/mlx4/intf.c | 13 ++++ 2 files changed, 67 insertions(+), 23 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 6debee494a16..1f8d0d2c5f17 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -2609,8 +2609,11 @@ static const struct ib_device_ops mlx4_ib_dev_fs_ops = { .destroy_flow = mlx4_ib_destroy_flow, }; -static void *mlx4_ib_add(struct mlx4_dev *dev) +static int mlx4_ib_probe(struct auxiliary_device *adev, + const struct auxiliary_device_id *id) { + struct mlx4_adev *madev = container_of(adev, struct mlx4_adev, adev); + struct mlx4_dev *dev = madev->mdev; struct mlx4_ib_dev *ibdev; int num_ports = 0; int i, j; @@ -2630,27 +2633,31 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) /* No point in registering a device with no ports... */ if (num_ports == 0) - return NULL; + return -ENODEV; ibdev = ib_alloc_device(mlx4_ib_dev, ib_dev); if (!ibdev) { dev_err(&dev->persist->pdev->dev, "Device struct alloc failed\n"); - return NULL; + return -ENOMEM; } iboe = &ibdev->iboe; - if (mlx4_pd_alloc(dev, &ibdev->priv_pdn)) + err = mlx4_pd_alloc(dev, &ibdev->priv_pdn); + if (err) goto err_dealloc; - if (mlx4_uar_alloc(dev, &ibdev->priv_uar)) + err = mlx4_uar_alloc(dev, &ibdev->priv_uar); + if (err) goto err_pd; ibdev->uar_map = ioremap((phys_addr_t) ibdev->priv_uar.pfn << PAGE_SHIFT, PAGE_SIZE); - if (!ibdev->uar_map) + if (!ibdev->uar_map) { + err = -ENOMEM; goto err_uar; + } MLX4_INIT_DOORBELL_LOCK(&ibdev->uar_lock); ibdev->dev = dev; @@ -2694,7 +2701,8 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) spin_lock_init(&iboe->lock); - if (init_node_data(ibdev)) + err = init_node_data(ibdev); + if (err) goto err_map; mlx4_init_sl2vl_tbl(ibdev); @@ -2726,6 +2734,7 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) new_counter_index = kmalloc(sizeof(*new_counter_index), GFP_KERNEL); if (!new_counter_index) { + err = -ENOMEM; if (allocated) mlx4_counter_free(ibdev->dev, counter_index); goto err_counter; @@ -2743,8 +2752,10 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) new_counter_index = kmalloc(sizeof(struct counter_index), GFP_KERNEL); - if (!new_counter_index) + if (!new_counter_index) { + err = -ENOMEM; goto err_counter; + } new_counter_index->index = counter_index; new_counter_index->allocated = 0; list_add_tail(&new_counter_index->list, @@ -2773,8 +2784,10 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) ibdev->ib_uc_qpns_bitmap = bitmap_alloc(ibdev->steer_qpn_count, GFP_KERNEL); - if (!ibdev->ib_uc_qpns_bitmap) + if (!ibdev->ib_uc_qpns_bitmap) { + err = -ENOMEM; goto err_steer_qp_release; + } if (dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_DMFS_IPOIB) { bitmap_zero(ibdev->ib_uc_qpns_bitmap, @@ -2794,17 +2807,21 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) for (j = 1; j <= ibdev->dev->caps.num_ports; j++) atomic64_set(&iboe->mac[j - 1], ibdev->dev->caps.def_mac[j]); - if (mlx4_ib_alloc_diag_counters(ibdev)) + err = mlx4_ib_alloc_diag_counters(ibdev); + if (err) goto err_steer_free_bitmap; - if (ib_register_device(&ibdev->ib_dev, "mlx4_%d", - &dev->persist->pdev->dev)) + err = ib_register_device(&ibdev->ib_dev, "mlx4_%d", + &dev->persist->pdev->dev); + if (err) goto err_diag_counters; - if (mlx4_ib_mad_init(ibdev)) + err = mlx4_ib_mad_init(ibdev); + if (err) goto err_reg; - if (mlx4_ib_init_sriov(ibdev)) + err = mlx4_ib_init_sriov(ibdev); + if (err) goto err_mad; if (!iboe->nb.notifier_call) { @@ -2844,7 +2861,8 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) err = mlx4_register_event_notifier(dev, &ibdev->mlx_nb); WARN(err, "failed to register mlx4 event notifier (%d)", err); - return ibdev; + auxiliary_set_drvdata(adev, ibdev); + return 0; err_notif: if (ibdev->iboe.nb.notifier_call) { @@ -2888,7 +2906,7 @@ static void *mlx4_ib_add(struct mlx4_dev *dev) err_dealloc: ib_dealloc_device(&ibdev->ib_dev); - return NULL; + return err; } int mlx4_ib_steer_qp_alloc(struct mlx4_ib_dev *dev, int count, int *qpn) @@ -2955,9 +2973,11 @@ int mlx4_ib_steer_qp_reg(struct mlx4_ib_dev *mdev, struct mlx4_ib_qp *mqp, return err; } -static void mlx4_ib_remove(struct mlx4_dev *dev, void *ibdev_ptr) +static void mlx4_ib_remove(struct auxiliary_device *adev) { - struct mlx4_ib_dev *ibdev = ibdev_ptr; + struct mlx4_adev *madev = container_of(adev, struct mlx4_adev, adev); + struct mlx4_dev *dev = madev->mdev; + struct mlx4_ib_dev *ibdev = auxiliary_get_drvdata(adev); int p; int i; @@ -3303,9 +3323,20 @@ static int mlx4_ib_event(struct notifier_block *this, unsigned long event, return NOTIFY_DONE; } -static struct mlx4_interface mlx4_ib_interface = { - .add = mlx4_ib_add, - .remove = mlx4_ib_remove, +static const struct auxiliary_device_id mlx4_ib_id_table[] = { + { .name = MLX4_ADEV_NAME ".ib" }, + {}, +}; + +MODULE_DEVICE_TABLE(auxiliary, mlx4_ib_id_table); + +static struct mlx4_adrv mlx4_ib_adrv = { + .adrv = { + .name = "ib", + .probe = mlx4_ib_probe, + .remove = mlx4_ib_remove, + .id_table = mlx4_ib_id_table, + }, .protocol = MLX4_PROT_IB_IPV6, .flags = MLX4_INTFF_BONDING }; @@ -3330,7 +3361,7 @@ static int __init mlx4_ib_init(void) if (err) goto clean_cm; - err = mlx4_register_interface(&mlx4_ib_interface); + err = mlx4_register_auxiliary_driver(&mlx4_ib_adrv); if (err) goto clean_mcg; @@ -3352,7 +3383,7 @@ static int __init mlx4_ib_init(void) static void __exit mlx4_ib_cleanup(void) { - mlx4_unregister_interface(&mlx4_ib_interface); + mlx4_unregister_auxiliary_driver(&mlx4_ib_adrv); mlx4_ib_mcg_destroy(); mlx4_ib_cm_destroy(); mlx4_ib_qp_event_cleanup(); diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 95ff0ea435e2..7579031786ac 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -59,11 +59,24 @@ static bool is_eth_supported(struct mlx4_dev *dev) return false; } +static bool is_ib_supported(struct mlx4_dev *dev) +{ + for (int port = 1; port <= dev->caps.num_ports; port++) + if (dev->caps.port_type[port] == MLX4_PORT_TYPE_IB) + return true; + + if (dev->caps.flags & MLX4_DEV_CAP_FLAG_IBOE) + return true; + + return false; +} + static const struct mlx4_adev_device { const char *suffix; bool (*is_supported)(struct mlx4_dev *dev); } mlx4_adev_devices[] = { { "eth", is_eth_supported }, + { "ib", is_ib_supported }, }; int mlx4_adev_init(struct mlx4_dev *dev) From patchwork Mon Aug 21 13:12:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Pavlu X-Patchwork-Id: 13359436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B9F7EE49B0 for ; Mon, 21 Aug 2023 13:12:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235312AbjHUNM7 (ORCPT ); Mon, 21 Aug 2023 09:12:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59418 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235299AbjHUNM5 (ORCPT ); Mon, 21 Aug 2023 09:12:57 -0400 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB9BCFF; Mon, 21 Aug 2023 06:12:49 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 872EC22183; Mon, 21 Aug 2023 13:12:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1692623563; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rbz4u7oCmPMNJcykidjw87NvSWL/0d2WcDQJ9atnf6U=; b=MBrQKC7AGsE8Y3bUEPoIL8B6H1Qyl+zDFp8IyKqJaM4FKZ1qmczAscraw/2+6OcMW+MuT3 u9+PNVQdGOod2zh+2Y1mGPBLYBAQwMSr/Rz+t592m9yElLhlCt9+dk4G+K5VL2W0kdJoF3 sT2oYSca9I+CKeYcUl2sUlC0eqjHl+k= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 3FE5D139BC; Mon, 21 Aug 2023 13:12:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id 2BGyDsti42QUVgAAMHmgww (envelope-from ); Mon, 21 Aug 2023 13:12:43 +0000 From: Petr Pavlu To: tariqt@nvidia.com, yishaih@nvidia.com, leon@kernel.org Cc: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, jgg@ziepe.ca, netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, Petr Pavlu , Leon Romanovsky Subject: [PATCH net-next v3 11/11] mlx4: Delete custom device management logic Date: Mon, 21 Aug 2023 15:12:25 +0200 Message-Id: <20230821131225.11290-12-petr.pavlu@suse.com> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230821131225.11290-1-petr.pavlu@suse.com> References: <20230821131225.11290-1-petr.pavlu@suse.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org After the conversion to use the auxiliary bus, the custom device management is not needed anymore and can be deleted. Signed-off-by: Petr Pavlu Tested-by: Leon Romanovsky Reviewed-by: Leon Romanovsky Acked-by: Tariq Toukan --- drivers/net/ethernet/mellanox/mlx4/intf.c | 125 ---------------------- drivers/net/ethernet/mellanox/mlx4/main.c | 28 ----- drivers/net/ethernet/mellanox/mlx4/mlx4.h | 3 - include/linux/mlx4/driver.h | 10 -- 4 files changed, 166 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx4/intf.c b/drivers/net/ethernet/mellanox/mlx4/intf.c index 7579031786ac..a371b970ac1e 100644 --- a/drivers/net/ethernet/mellanox/mlx4/intf.c +++ b/drivers/net/ethernet/mellanox/mlx4/intf.c @@ -38,15 +38,6 @@ #include "mlx4.h" -struct mlx4_device_context { - struct list_head list; - struct list_head bond_list; - struct mlx4_interface *intf; - void *context; -}; - -static LIST_HEAD(intf_list); -static LIST_HEAD(dev_list); static DEFINE_MUTEX(intf_mutex); static DEFINE_IDA(mlx4_adev_ida); @@ -156,77 +147,6 @@ static void del_adev(struct auxiliary_device *adev) auxiliary_device_uninit(adev); } -static void mlx4_add_device(struct mlx4_interface *intf, struct mlx4_priv *priv) -{ - struct mlx4_device_context *dev_ctx; - - dev_ctx = kmalloc(sizeof(*dev_ctx), GFP_KERNEL); - if (!dev_ctx) - return; - - dev_ctx->intf = intf; - dev_ctx->context = intf->add(&priv->dev); - - if (dev_ctx->context) { - spin_lock_irq(&priv->ctx_lock); - list_add_tail(&dev_ctx->list, &priv->ctx_list); - spin_unlock_irq(&priv->ctx_lock); - } else - kfree(dev_ctx); - -} - -static void mlx4_remove_device(struct mlx4_interface *intf, struct mlx4_priv *priv) -{ - struct mlx4_device_context *dev_ctx; - - list_for_each_entry(dev_ctx, &priv->ctx_list, list) - if (dev_ctx->intf == intf) { - spin_lock_irq(&priv->ctx_lock); - list_del(&dev_ctx->list); - spin_unlock_irq(&priv->ctx_lock); - - intf->remove(&priv->dev, dev_ctx->context); - kfree(dev_ctx); - return; - } -} - -int mlx4_register_interface(struct mlx4_interface *intf) -{ - struct mlx4_priv *priv; - - if (!intf->add || !intf->remove) - return -EINVAL; - - mutex_lock(&intf_mutex); - - list_add_tail(&intf->list, &intf_list); - list_for_each_entry(priv, &dev_list, dev_list) { - mlx4_add_device(intf, priv); - } - - mutex_unlock(&intf_mutex); - - return 0; -} -EXPORT_SYMBOL_GPL(mlx4_register_interface); - -void mlx4_unregister_interface(struct mlx4_interface *intf) -{ - struct mlx4_priv *priv; - - mutex_lock(&intf_mutex); - - list_for_each_entry(priv, &dev_list, dev_list) - mlx4_remove_device(intf, priv); - - list_del(&intf->list); - - mutex_unlock(&intf_mutex); -} -EXPORT_SYMBOL_GPL(mlx4_unregister_interface); - int mlx4_register_auxiliary_driver(struct mlx4_adrv *madrv) { return auxiliary_driver_register(&madrv->adrv); @@ -242,10 +162,7 @@ EXPORT_SYMBOL_GPL(mlx4_unregister_auxiliary_driver); int mlx4_do_bond(struct mlx4_dev *dev, bool enable) { struct mlx4_priv *priv = mlx4_priv(dev); - struct mlx4_device_context *dev_ctx = NULL, *temp_dev_ctx; - unsigned long flags; int i, ret; - LIST_HEAD(bond_list); if (!(dev->caps.flags2 & MLX4_DEV_CAP_FLAG2_PORT_REMAP)) return -EOPNOTSUPP; @@ -267,36 +184,6 @@ int mlx4_do_bond(struct mlx4_dev *dev, bool enable) dev->flags &= ~MLX4_FLAG_BONDED; } - spin_lock_irqsave(&priv->ctx_lock, flags); - list_for_each_entry_safe(dev_ctx, temp_dev_ctx, &priv->ctx_list, list) { - if (!(dev_ctx->intf->flags & MLX4_INTFF_BONDING)) - continue; - - if (mlx4_is_mfunc(dev)) { - mlx4_dbg(dev, - "SRIOV, disabled HA mode for intf proto %d\n", - dev_ctx->intf->protocol); - continue; - } - - list_add_tail(&dev_ctx->bond_list, &bond_list); - list_del(&dev_ctx->list); - } - spin_unlock_irqrestore(&priv->ctx_lock, flags); - - list_for_each_entry(dev_ctx, &bond_list, bond_list) { - dev_ctx->intf->remove(dev, dev_ctx->context); - dev_ctx->context = dev_ctx->intf->add(dev); - - spin_lock_irqsave(&priv->ctx_lock, flags); - list_add_tail(&dev_ctx->list, &priv->ctx_list); - spin_unlock_irqrestore(&priv->ctx_lock, flags); - - mlx4_dbg(dev, "Interface for protocol %d restarted with bonded mode %s\n", - dev_ctx->intf->protocol, enable ? - "enabled" : "disabled"); - } - mutex_lock(&intf_mutex); for (i = 0; i < ARRAY_SIZE(mlx4_adev_devices); i++) { @@ -447,16 +334,11 @@ static int rescan_drivers_locked(struct mlx4_dev *dev) int mlx4_register_device(struct mlx4_dev *dev) { - struct mlx4_priv *priv = mlx4_priv(dev); - struct mlx4_interface *intf; int ret; mutex_lock(&intf_mutex); dev->persist->interface_state |= MLX4_INTERFACE_STATE_UP; - list_add_tail(&priv->dev_list, &dev_list); - list_for_each_entry(intf, &intf_list, list) - mlx4_add_device(intf, priv); ret = rescan_drivers_locked(dev); @@ -474,9 +356,6 @@ int mlx4_register_device(struct mlx4_dev *dev) void mlx4_unregister_device(struct mlx4_dev *dev) { - struct mlx4_priv *priv = mlx4_priv(dev); - struct mlx4_interface *intf; - if (!(dev->persist->interface_state & MLX4_INTERFACE_STATE_UP)) return; @@ -495,10 +374,6 @@ void mlx4_unregister_device(struct mlx4_dev *dev) } mutex_lock(&intf_mutex); - list_for_each_entry(intf, &intf_list, list) - mlx4_remove_device(intf, priv); - - list_del(&priv->dev_list); dev->persist->interface_state &= ~MLX4_INTERFACE_STATE_UP; rescan_drivers_locked(dev); diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c index c4ec7377aa71..2581226836b5 100644 --- a/drivers/net/ethernet/mellanox/mlx4/main.c +++ b/drivers/net/ethernet/mellanox/mlx4/main.c @@ -42,7 +42,6 @@ #include #include #include -#include #include #include @@ -1091,27 +1090,6 @@ static int mlx4_slave_cap(struct mlx4_dev *dev) return err; } -static void mlx4_request_modules(struct mlx4_dev *dev) -{ - int port; - int has_ib_port = false; - int has_eth_port = false; -#define EN_DRV_NAME "mlx4_en" -#define IB_DRV_NAME "mlx4_ib" - - for (port = 1; port <= dev->caps.num_ports; port++) { - if (dev->caps.port_type[port] == MLX4_PORT_TYPE_IB) - has_ib_port = true; - else if (dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH) - has_eth_port = true; - } - - if (has_eth_port) - request_module_nowait(EN_DRV_NAME); - if (has_ib_port || (dev->caps.flags & MLX4_DEV_CAP_FLAG_IBOE)) - request_module_nowait(IB_DRV_NAME); -} - /* * Change the port configuration of the device. * Every user of this function must hold the port mutex. @@ -1147,7 +1125,6 @@ int mlx4_change_port_types(struct mlx4_dev *dev, mlx4_err(dev, "Failed to register device\n"); goto out; } - mlx4_request_modules(dev); } out: @@ -3426,9 +3403,6 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, devl_assert_locked(devlink); dev = &priv->dev; - INIT_LIST_HEAD(&priv->ctx_list); - spin_lock_init(&priv->ctx_lock); - err = mlx4_adev_init(dev); if (err) return err; @@ -3732,8 +3706,6 @@ static int mlx4_load_one(struct pci_dev *pdev, int pci_dev_data, if (err) goto err_port; - mlx4_request_modules(dev); - mlx4_sense_init(dev); mlx4_start_sense(dev); diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h index 36de79c8d053..d7d856d1758a 100644 --- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h +++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h @@ -882,9 +882,6 @@ enum { struct mlx4_priv { struct mlx4_dev dev; - struct list_head dev_list; - struct list_head ctx_list; - spinlock_t ctx_lock; struct mlx4_adev **adev; int adev_idx; struct atomic_notifier_head event_nh; diff --git a/include/linux/mlx4/driver.h b/include/linux/mlx4/driver.h index 9cf157d381c6..69825223081f 100644 --- a/include/linux/mlx4/driver.h +++ b/include/linux/mlx4/driver.h @@ -58,22 +58,12 @@ enum { MLX4_INTFF_BONDING = 1 << 0 }; -struct mlx4_interface { - void * (*add) (struct mlx4_dev *dev); - void (*remove)(struct mlx4_dev *dev, void *context); - struct list_head list; - enum mlx4_protocol protocol; - int flags; -}; - struct mlx4_adrv { struct auxiliary_driver adrv; enum mlx4_protocol protocol; int flags; }; -int mlx4_register_interface(struct mlx4_interface *intf); -void mlx4_unregister_interface(struct mlx4_interface *intf); int mlx4_register_auxiliary_driver(struct mlx4_adrv *madrv); void mlx4_unregister_auxiliary_driver(struct mlx4_adrv *madrv);