From patchwork Thu Nov 12 19:24:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Parav Pandit X-Patchwork-Id: 11901269 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E51961391 for ; Thu, 12 Nov 2020 19:25:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B599A2222F for ; Thu, 12 Nov 2020 19:25:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=nvidia.com header.i=@nvidia.com header.b="OZk/cVCJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726950AbgKLTZD (ORCPT ); Thu, 12 Nov 2020 14:25:03 -0500 Received: from hqnvemgate24.nvidia.com ([216.228.121.143]:11377 "EHLO hqnvemgate24.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726930AbgKLTZB (ORCPT ); Thu, 12 Nov 2020 14:25:01 -0500 Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate24.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Thu, 12 Nov 2020 11:25:09 -0800 Received: from sw-mtx-036.mtx.labs.mlnx (172.20.13.39) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Thu, 12 Nov 2020 19:25:00 +0000 From: Parav Pandit To: , , CC: , , , , , , , Parav Pandit , Vu Pham Subject: [PATCH net-next 13/13] net/mlx5: SF, Port function state change support Date: Thu, 12 Nov 2020 21:24:23 +0200 Message-ID: <20201112192424.2742-14-parav@nvidia.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201112192424.2742-1-parav@nvidia.com> References: <20201112192424.2742-1-parav@nvidia.com> MIME-Version: 1.0 X-Originating-IP: [172.20.13.39] X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To HQMAIL107.nvidia.com (172.20.187.13) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1605209109; bh=BPkdJ1ifA/bO+m9fBGi+nxds8HnjhAL3G/KQIhHvcAk=; h=From:To:CC:Subject:Date:Message-ID:X-Mailer:In-Reply-To: References:MIME-Version:Content-Transfer-Encoding:Content-Type: X-Originating-IP:X-ClientProxiedBy; b=OZk/cVCJmKcI2fb+0Z1Zi/h9ZLrga56KrZ1PPsPA/XOoZquIfCXKE3Yb89P4C+3Sj rvdYa/mgRK0gJTLj7pKFgL6tNhQvIcOkYm4f5W1m1w6GjD4zpDOoHTubxSgrxohRVG A0kmfjVR0TYRR+37fx93TK2GApcAXIHPs6My0knGuYne0okLUMkT5b9LG2LzRTkCLb xjI+GrOZ7J/f2YaD9Q/X9YdEUkygeEWfA00HFIcXUWsk0sKnehWtg79xc5jKwgvSpA +JaWRtdJ/z3g8IzTK2d7qEYcFqWPbVo+ocW0Sw2pE2lzNUhWRBgGM7IjdsS9SoH02G 1g4oaVJdekcdg== Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Support changing the state of the SF port's function through devlink. When activating the SF port's function, enable the hca in the device followed by adding its auxiliary device. When deactivating the SF port's function, delete its auxiliary device followed by disabling the HCA. Port function attributes get/set callbacks are invoked with devlink instance lock held. Such callbacks need to synchronize with sf port table getting disabled. These callbacks while operating on the devlink port, synchronize with table disable context by holding table refcount. $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev $ devlink port show pci/0000:06:00.0/65535: type eth netdev ens2f0np0 flavour physical port 0 splittable false $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88 $ devlink port show ens2f0npf0sf88 pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false function: hw_addr 00:00:00:00:88:88 state inactive opstate detached $ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88 state active $ devlink port show ens2f0npf0sf88 -jp { "port": { "pci/0000:06:00.0/32768": { "type": "eth", "netdev": "ens2f0npf0sf88", "flavour": "pcisf", "controller": 0, "pfnum": 0, "sfnum": 88, "external": false, "splittable": false, "function": { "hw_addr": "00:00:00:00:88:88", "state": "active", "opstate": "attached" } } } } On port function activation, an auxiliary device is created in below example. $ devlink dev show devlink dev show auxiliary/mlx5_core.sf.0 $ devlink port show auxiliary/mlx5_core.sf.0/1 auxiliary/mlx5_core.sf.0/1: type eth netdev p0sf88 flavour virtual port 0 splittable false Signed-off-by: Parav Pandit Reviewed-by: Vu Pham --- .../net/ethernet/mellanox/mlx5/core/devlink.c | 2 + .../net/ethernet/mellanox/mlx5/core/sf/sf.c | 128 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/sf/sf.h | 7 + 3 files changed, 137 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c index 7ad8dc26cb74..22d22959e6f6 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -192,6 +192,8 @@ static const struct devlink_ops mlx5_devlink_ops = { #ifdef CONFIG_MLX5_SF_MANAGER .port_new = mlx5_devlink_sf_port_new, .port_del = mlx5_devlink_sf_port_del, + .port_function_state_get = mlx5_devlink_sf_port_fn_state_get, + .port_function_state_set = mlx5_devlink_sf_port_fn_state_set, #endif .flash_update = mlx5_devlink_flash_update, .info_get = mlx5_devlink_info_get, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/sf/sf.c b/drivers/net/ethernet/mellanox/mlx5/core/sf/sf.c index dff44ab5057d..7e90629fd910 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/sf/sf.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/sf/sf.c @@ -4,6 +4,7 @@ #include #include "eswitch.h" #include "priv.h" +#include "sf/dev/dev.h" struct mlx5_sf { struct devlink_port dl_port; @@ -11,6 +12,7 @@ struct mlx5_sf { u32 usr_sfnum; u16 sw_id; u16 hw_fn_id; + enum devlink_port_function_state state; }; struct mlx5_sf_table { @@ -115,6 +117,7 @@ mlx5_sf_alloc(struct mlx5_sf_table *table, u32 sfnum, struct netlink_ext_ack *ex if (err) goto id_err; + sf->state = DEVLINK_PORT_FUNCTION_STATE_INACTIVE; dl_port_index = mlx5_esw_vport_to_devlink_port_index(table->dev, sf->hw_fn_id); sf->port_index = dl_port_index; sf->usr_sfnum = sfnum; @@ -156,6 +159,126 @@ static void mlx5_sf_table_put(struct mlx5_sf_table *table) complete(&table->disable_complete); } +static int +mlx5_sf_state_get(struct mlx5_core_dev *dev, struct mlx5_sf *sf, + enum devlink_port_function_state *state, + enum devlink_port_function_opstate *opstate) +{ + int err = 0; + + *state = sf->state; + switch (sf->state) { + case DEVLINK_PORT_FUNCTION_STATE_ACTIVE: + *opstate = DEVLINK_PORT_FUNCTION_OPSTATE_ATTACHED; + break; + case DEVLINK_PORT_FUNCTION_STATE_INACTIVE: + *opstate = DEVLINK_PORT_FUNCTION_OPSTATE_DETACHED; + break; + default: + err = -EINVAL; + break; + } + return err; +} + +int mlx5_devlink_sf_port_fn_state_get(struct devlink *devlink, struct devlink_port *dl_port, + enum devlink_port_function_state *state, + enum devlink_port_function_opstate *opstate, + struct netlink_ext_ack *extack) +{ + struct mlx5_core_dev *dev = devlink_priv(devlink); + struct mlx5_sf_table *table; + int err = -EOPNOTSUPP; + struct mlx5_sf *sf; + + table = mlx5_sf_table_try_get(dev); + if (!table) + return -EOPNOTSUPP; + + sf = mlx5_sf_lookup_by_index(table, dl_port->index); + if (!sf) + goto sf_err; + err = mlx5_sf_state_get(dev, sf, state, opstate); +sf_err: + mlx5_sf_table_put(table); + return err; +} + +static int mlx5_sf_activate(struct mlx5_core_dev *dev, struct mlx5_sf *sf) +{ + int err; + + err = mlx5_cmd_sf_enable_hca(dev, sf->hw_fn_id); + if (err) + return err; + + err = mlx5_sf_dev_add(dev, sf->sw_id, sf->usr_sfnum); + if (err) + goto dev_err; + + sf->state = DEVLINK_PORT_FUNCTION_STATE_ACTIVE; + return 0; + +dev_err: + mlx5_cmd_sf_disable_hca(dev, sf->hw_fn_id); + return err; +} + +static int mlx5_sf_deactivate(struct mlx5_core_dev *dev, struct mlx5_sf *sf) +{ + int err; + + mlx5_sf_dev_del(dev, sf->sw_id); + err = mlx5_cmd_sf_disable_hca(dev, sf->hw_fn_id); + if (err) + return err; + sf->state = DEVLINK_PORT_FUNCTION_STATE_INACTIVE; + return 0; +} + +static int mlx5_sf_state_set(struct mlx5_core_dev *dev, struct mlx5_sf *sf, + enum devlink_port_function_state state) +{ + int err; + + if (sf->state == state) + return 0; + if (state == DEVLINK_PORT_FUNCTION_STATE_ACTIVE) + err = mlx5_sf_activate(dev, sf); + else if (state == DEVLINK_PORT_FUNCTION_STATE_INACTIVE) + err = mlx5_sf_deactivate(dev, sf); + else + err = -EINVAL; + return err; +} + +int mlx5_devlink_sf_port_fn_state_set(struct devlink *devlink, struct devlink_port *dl_port, + enum devlink_port_function_state state, + struct netlink_ext_ack *extack) +{ + struct mlx5_core_dev *dev = devlink_priv(devlink); + struct mlx5_sf_table *table; + struct mlx5_sf *sf; + int err; + + table = mlx5_sf_table_try_get(dev); + if (!table) { + NL_SET_ERR_MSG_MOD(extack, + "Port state set is only supported in eswitch switchdev mode or SF ports are disabled."); + return -EOPNOTSUPP; + } + sf = mlx5_sf_lookup_by_index(table, dl_port->index); + if (!sf) { + err = -ENODEV; + goto out; + } + + err = mlx5_sf_state_set(dev, sf, state); +out: + mlx5_sf_table_put(table); + return err; +} + static int mlx5_sf_add(struct mlx5_core_dev *dev, struct mlx5_sf_table *table, const struct devlink_port_new_attrs *new_attr, struct netlink_ext_ack *extack) @@ -184,6 +307,10 @@ static void mlx5_sf_del(struct mlx5_core_dev *dev, struct mlx5_sf_table *table, { struct mlx5_eswitch *esw = dev->priv.eswitch; + if (sf->state == DEVLINK_PORT_FUNCTION_STATE_ACTIVE) { + mlx5_sf_dev_del(dev, sf->sw_id); + mlx5_cmd_sf_disable_hca(dev, sf->hw_fn_id); + } mlx5_esw_offloads_sf_vport_disable(esw, sf->hw_fn_id); mlx5_sf_free(table, sf); } @@ -343,6 +470,7 @@ int mlx5_sf_table_init(struct mlx5_core_dev *dev) table->dev = dev; ida_init(&table->fn_ida); + refcount_set(&table->refcount, 0); dev->priv.sf_table = table; table->esw_nb.notifier_call = mlx5_sf_esw_event; err = mlx5_esw_event_notifier_register(dev->priv.eswitch, &table->esw_nb); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h b/drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h index 555b19a5880d..3d1c459b9936 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h @@ -25,6 +25,13 @@ int mlx5_devlink_sf_port_new(struct devlink *devlink, const struct devlink_port_ struct netlink_ext_ack *extack); int mlx5_devlink_sf_port_del(struct devlink *devlink, unsigned int port_index, struct netlink_ext_ack *extack); +int mlx5_devlink_sf_port_fn_state_get(struct devlink *devlink, struct devlink_port *dl_port, + enum devlink_port_function_state *state, + enum devlink_port_function_opstate *opstate, + struct netlink_ext_ack *extack); +int mlx5_devlink_sf_port_fn_state_set(struct devlink *devlink, struct devlink_port *dl_port, + enum devlink_port_function_state state, + struct netlink_ext_ack *extack); #else