From patchwork Thu Nov 12 19:24:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Parav Pandit X-Patchwork-Id: 11901279 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BE05C16C1 for ; Thu, 12 Nov 2020 19:25:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D0F82222F for ; Thu, 12 Nov 2020 19:25:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="Lvx7sg6o" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726998AbgKLTZP (ORCPT ); Thu, 12 Nov 2020 14:25:15 -0500 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:11882 "EHLO hqnvemgate25.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726826AbgKLTYz (ORCPT ); Thu, 12 Nov 2020 14:24:55 -0500 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Thu, 12 Nov 2020 11:24:49 -0800 Received: from sw-mtx-036.mtx.labs.mlnx (172.20.13.39) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 12 Nov 2020 19:24:53 +0000 From: Parav Pandit To: , , CC: , , , , , , , Parav Pandit , Vu Pham Subject: [PATCH net-next 07/13] net/mlx5: SF, Add auxiliary device support Date: Thu, 12 Nov 2020 21:24:17 +0200 Message-ID: <20201112192424.2742-8-parav@nvidia.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201112192424.2742-1-parav@nvidia.com> References: <20201112192424.2742-1-parav@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.13.39] X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To HQMAIL107.nvidia.com (172.20.187.13) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1605209089; bh=XkRd9aq6/G3DKokOFC93KLRgF4gaeOp5YmQLYKMRFu0=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:MIME-Version:Content-Transfer-Encoding:Content-Type: X-Originating-IP:X-ClientProxiedBy; b=Lvx7sg6osG76sqP2wNoIt2Kpa2a5aQmfvt6wVxk3yH9l30z5PhAaaND/1VX21GqTN eMuPxnIdi5aObmkrtZsyXNYwa3Fe4PznzHiADdNwcI0ffjsuh6jd+Y1047KWLv9KNJ 8ENQY4zUHjObyipcexWEUSf8iKAxQ9nD96FWT6FZHp9KYrpgSZqek9GDItovINlhrH TQrPgxGru8VJM62cwzEDbHfj2CNHoqae6Xf+lzcwPqBl5xZNL+d8bic3Vcq4ZplbO4 RjM5FhMe5qYrhbHK5RUSecDmmsoQ8ivGa3ZzvlRog9bi7FIFPIyIxND3vP7HcWm7IG UZxyE2UZii0sw== Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Introduce API to add and delete an auxiliary device for an SF. Each SF has its own dedicated window in the PCI BAR 2. SF device is similar to PCI PF and VF that supports multiple class of devices such as net, rdma and vdpa. SF device will be added or removed in subsequent patch during SF devlink port function state change command. A subfunction device exposes user supplied subfunction number which will be further used by systemd/udev to have deterministic name for its netdevice and rdma device. An mlx5 subfunction auxiliary device example: $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev $ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88 $ devlink port show ens2f0npf0sf88 pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false function: hw_addr 00:00:00:00:88:88 state inactive opstate detached $ devlink port function set ens2f0npf0sf88 hw_addr 00:00:00:00:88:88 state active On activation, $ ls -l /sys/bus/auxiliary/devices/ mlx5_core.sf.0 -> ../../../devices/pci0000:00/0000:00:03.0/0000:06:00.0/mlx5_core.sf.0 $ cat /sys/bus/auxiliary/devices/mlx5_core.sf.0/sfnum 88 Signed-off-by: Parav Pandit Reviewed-by: Vu Pham --- .../net/ethernet/mellanox/mlx5/core/Kconfig | 9 + .../net/ethernet/mellanox/mlx5/core/Makefile | 4 + .../net/ethernet/mellanox/mlx5/core/main.c | 12 + .../ethernet/mellanox/mlx5/core/sf/dev/dev.c | 213 ++++++++++++++++++ .../ethernet/mellanox/mlx5/core/sf/dev/dev.h | 55 +++++ include/linux/mlx5/driver.h | 4 + 6 files changed, 297 insertions(+) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig index 485478979b1a..10dfaf671c90 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Kconfig +++ b/drivers/net/ethernet/mellanox/mlx5/core/Kconfig @@ -202,3 +202,12 @@ config MLX5_SW_STEERING default y help Build support for software-managed steering in the NIC. + +config MLX5_SF + bool "Mellanox Technologies subfunction device support using auxiliary device" + depends on MLX5_CORE && MLX5_CORE_EN + default n + help + Build support for subfuction device in the NIC. A Mellanox subfunction + device can support RDMA, netdevice and vdpa device. + It is similar to a SRIOV VF but it doesn't require SRIOV support. diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile index 2d477f9a8cb7..ee866da1d9ba 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile @@ -85,3 +85,7 @@ mlx5_core-$(CONFIG_MLX5_SW_STEERING) += steering/dr_domain.o steering/dr_table.o steering/dr_ste.o steering/dr_send.o \ steering/dr_cmd.o steering/dr_fw.o \ steering/dr_action.o steering/fs_dr.o +# +# SF device +# +mlx5_core-$(CONFIG_MLX5_SF) += sf/dev/dev.o diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index 37fa56904235..a1ba6056952b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -73,6 +73,7 @@ #include "ecpf.h" #include "lib/hv_vhca.h" #include "diag/rsc_dump.h" +#include "sf/dev/dev.h" MODULE_AUTHOR("Eli Cohen "); MODULE_DESCRIPTION("Mellanox 5th generation network adapters (ConnectX series) core driver"); @@ -884,6 +885,12 @@ static int mlx5_init_once(struct mlx5_core_dev *dev) goto err_eswitch_cleanup; } + err = mlx5_sf_dev_table_init(dev); + if (err) { + mlx5_core_err(dev, "Failed to init SF device table %d\n", err); + goto err_sf_dev_cleanup; + } + dev->dm = mlx5_dm_create(dev); if (IS_ERR(dev->dm)) mlx5_core_warn(dev, "Failed to init device memory%d\n", err); @@ -894,6 +901,8 @@ static int mlx5_init_once(struct mlx5_core_dev *dev) return 0; +err_sf_dev_cleanup: + mlx5_fpga_cleanup(dev); err_eswitch_cleanup: mlx5_eswitch_cleanup(dev->priv.eswitch); err_sriov_cleanup: @@ -925,6 +934,7 @@ static void mlx5_cleanup_once(struct mlx5_core_dev *dev) mlx5_hv_vhca_destroy(dev->hv_vhca); mlx5_fw_tracer_destroy(dev->tracer); mlx5_dm_cleanup(dev); + mlx5_sf_dev_table_cleanup(dev); mlx5_fpga_cleanup(dev); mlx5_eswitch_cleanup(dev->priv.eswitch); mlx5_sriov_cleanup(dev); @@ -1141,6 +1151,7 @@ static int mlx5_load(struct mlx5_core_dev *dev) goto err_ec; } + mlx5_sf_dev_table_create(dev); return 0; err_ec: @@ -1171,6 +1182,7 @@ static int mlx5_load(struct mlx5_core_dev *dev) static void mlx5_unload(struct mlx5_core_dev *dev) { + mlx5_sf_dev_table_destroy(dev); mlx5_ec_cleanup(dev); mlx5_sriov_detach(dev); mlx5_cleanup_fs(dev); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c b/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c new file mode 100644 index 000000000000..a25f6027b7cd --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.c @@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2020 Mellanox Technologies Ltd */ + +#include +#include +#include "mlx5_core.h" +#include "dev.h" + +struct mlx5_sf_dev_table { + /* Serializes table access between driver unload context and + * device add/remove user command context. + */ + struct mutex table_lock; + struct xarray devices; + unsigned int max_sfs; + phys_addr_t base_address; + u64 sf_bar_length; +}; + +static bool mlx5_sf_dev_supported(const struct mlx5_core_dev *dev) +{ + return MLX5_CAP_GEN(dev, sf); +} + +static ssize_t sfnum_show(struct device *dev, struct device_attribute *attr, char *buf) +{ + struct auxiliary_device *adev = container_of(dev, struct auxiliary_device, dev); + struct mlx5_sf_dev *sf_dev = container_of(adev, struct mlx5_sf_dev, adev); + + return scnprintf(buf, PAGE_SIZE, "%u\n", sf_dev->sfnum); +} +static DEVICE_ATTR_RO(sfnum); + +static struct attribute *sf_device_attrs[] = { + &dev_attr_sfnum.attr, + NULL, +}; + +static const struct attribute_group sf_attr_group = { + .attrs = sf_device_attrs, +}; + +static const struct attribute_group *sf_attr_groups[2] = { + &sf_attr_group, + NULL +}; + +static void mlx5_sf_dev_release(struct device *device) +{ + struct auxiliary_device *adev = container_of(device, struct auxiliary_device, dev); + struct mlx5_sf_dev *sf_dev = container_of(adev, struct mlx5_sf_dev, adev); + + mlx5_adev_idx_free(sf_dev->adev.id); + kfree(sf_dev); +} + +static void mlx5_sf_dev_remove(struct mlx5_sf_dev *sf_dev) +{ + auxiliary_device_delete(&sf_dev->adev); + auxiliary_device_uninit(&sf_dev->adev); +} + +int mlx5_sf_dev_add(struct mlx5_core_dev *dev, u16 sf_index, u32 sfnum) +{ + struct mlx5_sf_dev_table *table = dev->priv.sf_dev_table; + struct mlx5_sf_dev *sf_dev; + struct pci_dev *pdev; + int id; + int err; + + id = mlx5_adev_idx_alloc(); + if (id < 0) + return id; + + sf_dev = kzalloc(sizeof(*sf_dev), GFP_KERNEL); + if (!sf_dev) { + mlx5_adev_idx_free(id); + return -ENOMEM; + } + pdev = dev->pdev; + sf_dev->adev.id = id; + sf_dev->adev.name = MLX5_SF_DEV_ID_NAME; + sf_dev->adev.dev.release = mlx5_sf_dev_release; + sf_dev->adev.dev.parent = &pdev->dev; + sf_dev->adev.dev.groups = sf_attr_groups; + sf_dev->sfnum = sfnum; + sf_dev->parent_mdev = dev; + + /* Serialize with unloading the driver. */ + mutex_lock(&table->table_lock); + if (!table->max_sfs) { + mlx5_adev_idx_free(id); + kfree(sf_dev); + err = -EOPNOTSUPP; + goto add_err; + } + sf_dev->bar_base_addr = table->base_address + (sf_index * table->sf_bar_length); + + err = auxiliary_device_init(&sf_dev->adev); + if (err) { + mlx5_adev_idx_free(id); + kfree(sf_dev); + goto add_err; + } + + err = auxiliary_device_add(&sf_dev->adev); + if (err) { + put_device(&sf_dev->adev.dev); + goto add_err; + } + + err = xa_insert(&table->devices, sf_index, sf_dev, GFP_KERNEL); + if (err) + goto xa_err; + mutex_unlock(&table->table_lock); + return 0; + +xa_err: + mlx5_sf_dev_remove(sf_dev); +add_err: + mutex_unlock(&table->table_lock); + return err; +} + +void mlx5_sf_dev_del(struct mlx5_core_dev *dev, u16 sf_index) +{ + struct mlx5_sf_dev_table *table = dev->priv.sf_dev_table; + struct mlx5_sf_dev *sf_dev; + + mutex_lock(&table->table_lock); + sf_dev = xa_load(&table->devices, sf_index); + if (!sf_dev) + goto done; + + xa_erase(&table->devices, sf_index); + mlx5_sf_dev_remove(sf_dev); +done: + mutex_unlock(&table->table_lock); +} + +int mlx5_sf_dev_table_init(struct mlx5_core_dev *dev) +{ + struct mlx5_sf_dev_table *table; + + if (!mlx5_sf_dev_supported(dev)) + return 0; + + table = kzalloc(sizeof(*table), GFP_KERNEL); + if (!table) + return -ENOMEM; + + mutex_init(&table->table_lock); + xa_init(&table->devices); + dev->priv.sf_dev_table = table; + return 0; +} + +void mlx5_sf_dev_table_cleanup(struct mlx5_core_dev *dev) +{ + struct mlx5_sf_dev_table *table = dev->priv.sf_dev_table; + + if (!table) + return; + + WARN_ON(!xa_empty(&table->devices)); + mutex_destroy(&table->table_lock); + kfree(table); + dev->priv.sf_dev_table = NULL; +} + +void mlx5_sf_dev_table_create(struct mlx5_core_dev *dev) +{ + struct mlx5_sf_dev_table *table = dev->priv.sf_dev_table; + unsigned int max_sfs; + + if (!table) + return; + + /* Honor the caps changed during reload */ + if (!mlx5_sf_dev_supported(dev)) + return; + + max_sfs = 1 << MLX5_CAP_GEN(dev, log_max_sf); + table->base_address = pci_resource_start(dev->pdev, 2); + table->sf_bar_length = 1 << (MLX5_CAP_GEN(dev, log_min_sf_size) + 12); + mutex_lock(&table->table_lock); + table->max_sfs = max_sfs; + mutex_unlock(&table->table_lock); +} + +static void mlx5_sf_dev_destroy_all(struct mlx5_sf_dev_table *table) +{ + struct mlx5_sf_dev *sf_dev; + unsigned long index; + + xa_for_each(&table->devices, index, sf_dev) { + xa_erase(&table->devices, index); + mlx5_sf_dev_remove(sf_dev); + } +} + +void mlx5_sf_dev_table_destroy(struct mlx5_core_dev *dev) +{ + struct mlx5_sf_dev_table *table = dev->priv.sf_dev_table; + + if (!table) + return; + + mutex_lock(&table->table_lock); + table->max_sfs = 0; + mlx5_sf_dev_destroy_all(table); + mutex_unlock(&table->table_lock); +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h b/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h new file mode 100644 index 000000000000..d81612122a45 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/sf/dev/dev.h @@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2020 Mellanox Technologies Ltd */ + +#ifndef __MLX5_SF_DEV_H__ +#define __MLX5_SF_DEV_H__ + +#ifdef CONFIG_MLX5_SF + +#include + +#define MLX5_SF_DEV_ID_NAME "sf" + +struct mlx5_sf_dev { + struct auxiliary_device adev; + struct mlx5_core_dev *parent_mdev; + phys_addr_t bar_base_addr; + u32 sfnum; +}; + +void __exit mlx5_sf_dev_exit(void); +int mlx5_sf_dev_table_init(struct mlx5_core_dev *dev); +void mlx5_sf_dev_table_cleanup(struct mlx5_core_dev *dev); +void mlx5_sf_dev_table_create(struct mlx5_core_dev *dev); +void mlx5_sf_dev_table_destroy(struct mlx5_core_dev *dev); + +int mlx5_sf_dev_add(struct mlx5_core_dev *dev, u16 sf_index, u32 sfnum); +void mlx5_sf_dev_del(struct mlx5_core_dev *dev, u16 sf_index); + +#else + +static inline void __exit mlx5_sf_dev_exit(void) +{ +} + +static inline int mlx5_sf_dev_table_init(struct mlx5_core_dev *dev) +{ + return 0; +} + +static inline void mlx5_sf_dev_table_cleanup(struct mlx5_core_dev *dev) +{ +} + +static inline int mlx5_sf_dev_table_create(struct mlx5_core_dev *dev) +{ + return 0; +} + +static inline void mlx5_sf_dev_table_destroy(struct mlx5_core_dev *dev) +{ +} + +#endif + +#endif diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 28e9b2f17eb9..151cacab07db 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -507,6 +507,7 @@ struct mlx5_devcom; struct mlx5_fw_reset; struct mlx5_eq_table; struct mlx5_irq_table; +struct mlx5_sf_dev_table; struct mlx5_rate_limit { u32 rate; @@ -603,6 +604,9 @@ struct mlx5_priv { struct mlx5_bfreg_data bfregs; struct mlx5_uars_page *uar; +#ifdef CONFIG_MLX5_SF + struct mlx5_sf_dev_table *sf_dev_table; +#endif }; enum mlx5_device_state {