Message ID | 20220418064241.2925668-1-idosch@nvidia.com (mailing list archive) |
---|---|
Headers | show |
Series | Introduce line card support for modular switch | expand |
Hello: This series was applied to netdev/net-next.git (master) by David S. Miller <davem@davemloft.net>: On Mon, 18 Apr 2022 09:42:24 +0300 you wrote: > Jiri says: > > This patchset introduces support for modular switch systems and also > introduces mlxsw support for NVIDIA Mellanox SN4800 modular switch. > It contains 8 slots to accommodate line cards - replaceable PHY modules > which may contain gearboxes. > Currently supported line card: > 16X 100GbE (QSFP28) > Other line cards that are going to be supported: > 8X 200GbE (QSFP56) > 4X 400GbE (QSFP-DD) > There may be other types of line cards added in the future. > > [...] Here is the summary with links: - [net-next,01/17] devlink: add support to create line card and expose to user https://git.kernel.org/netdev/net-next/c/c246f9b5fd61 - [net-next,02/17] devlink: implement line card provisioning https://git.kernel.org/netdev/net-next/c/fcdc8ce23a30 - [net-next,03/17] devlink: implement line card active state https://git.kernel.org/netdev/net-next/c/fc9f50d5b366 - [net-next,04/17] devlink: add port to line card relationship set https://git.kernel.org/netdev/net-next/c/b83758598538 - [net-next,05/17] mlxsw: spectrum: Allow lane to start from non-zero index https://git.kernel.org/netdev/net-next/c/bac62191a3d4 - [net-next,06/17] mlxsw: spectrum: Allocate port mapping array of structs instead of pointers https://git.kernel.org/netdev/net-next/c/d3ad2d88209f - [net-next,07/17] mlxsw: reg: Add Ports Mapping Event Configuration Register https://git.kernel.org/netdev/net-next/c/ebf0c5341731 - [net-next,08/17] mlxsw: Narrow the critical section of devl_lock during ports creation/removal https://git.kernel.org/netdev/net-next/c/adc6462376b1 - [net-next,09/17] mlxsw: spectrum: Introduce port mapping change event processing https://git.kernel.org/netdev/net-next/c/b0ec003e9a90 - [net-next,10/17] mlxsw: reg: Add Management DownStream Device Query Register https://git.kernel.org/netdev/net-next/c/505f524dc660 - [net-next,11/17] mlxsw: reg: Add Management DownStream Device Control Register https://git.kernel.org/netdev/net-next/c/5290a8ff2e11 - [net-next,12/17] mlxsw: reg: Add Management Binary Code Transfer Register https://git.kernel.org/netdev/net-next/c/5bade5aa4afc - [net-next,13/17] mlxsw: core_linecards: Add line card objects and implement provisioning https://git.kernel.org/netdev/net-next/c/b217127e5e4e - [net-next,14/17] mlxsw: core_linecards: Implement line card activation process https://git.kernel.org/netdev/net-next/c/ee7a70fa671b - [net-next,15/17] mlxsw: core: Extend driver ops by remove selected ports op https://git.kernel.org/netdev/net-next/c/45bf3b7267e0 - [net-next,16/17] mlxsw: spectrum: Add port to linecard mapping https://git.kernel.org/netdev/net-next/c/6445eef0f600 - [net-next,17/17] selftests: mlxsw: Introduce devlink line card provision/unprovision/activation tests https://git.kernel.org/netdev/net-next/c/e1fad9517f0f You are awesome, thank you!
On 4/18/22 12:42 AM, Ido Schimmel wrote: > Jiri says: > > This patchset introduces support for modular switch systems and also > introduces mlxsw support for NVIDIA Mellanox SN4800 modular switch. > It contains 8 slots to accommodate line cards - replaceable PHY modules > which may contain gearboxes. > Currently supported line card: > 16X 100GbE (QSFP28) > Other line cards that are going to be supported: > 8X 200GbE (QSFP56) > 4X 400GbE (QSFP-DD) > There may be other types of line cards added in the future. > > To be consistent with the port split configuration (splitter cabels), > the line card entities are treated in the similar way. The nature of > a line card is not "a pluggable device", but "a pluggable PHY module". > > A concept of "provisioning" is introduced. The user may "provision" > certain slot with a line card type. Driver then creates all instances > (devlink ports, netdevices, etc) related to this line card type. It does > not matter if the line card is plugged-in at the time. User is able to > configure netdevices, devlink ports, setup port splitters, etc. From the > perspective of the switch ASIC, all is present and can be configured. > > The carrier of netdevices stays down if the line card is not plugged-in. > Once the line card is inserted and activated, the carrier of > the related netdevices is then reflecting the physical line state, > same as for an ordinary fixed port. > > Once user does not want to use the line card related instances > anymore, he can "unprovision" the slot. Driver then removes the > instances. > > Patches 1-4 are extending devlink driver API and UAPI in order to > register, show, dump, provision and activate the line card. > Patches 5-17 are implementing the introduced API in mlxsw. > The last patch adds a selftest for mlxsw line cards. > > Example: > $ devlink port # No ports are listed > $ devlink lc > pci/0000:01:00.0: > lc 1 state unprovisioned > supported_types: > 16x100G > lc 2 state unprovisioned > supported_types: > 16x100G > lc 3 state unprovisioned > supported_types: > 16x100G > lc 4 state unprovisioned > supported_types: > 16x100G > lc 5 state unprovisioned > supported_types: > 16x100G > lc 6 state unprovisioned > supported_types: > 16x100G > lc 7 state unprovisioned > supported_types: > 16x100G > lc 8 state unprovisioned > supported_types: > 16x100G > > Note that driver exposes list supported line card types. Currently > there is only one: "16x100G". > > To provision the slot #8: > > $ devlink lc set pci/0000:01:00.0 lc 8 type 16x100G > $ devlink lc show pci/0000:01:00.0 lc 8 > pci/0000:01:00.0: > lc 8 state active type 16x100G > supported_types: > 16x100G > $ devlink port > pci/0000:01:00.0/0: type notset flavour cpu port 0 splittable false > pci/0000:01:00.0/53: type eth netdev enp1s0nl8p1 flavour physical lc 8 port 1 splittable true lanes 4 > pci/0000:01:00.0/54: type eth netdev enp1s0nl8p2 flavour physical lc 8 port 2 splittable true lanes 4 > pci/0000:01:00.0/55: type eth netdev enp1s0nl8p3 flavour physical lc 8 port 3 splittable true lanes 4 > pci/0000:01:00.0/56: type eth netdev enp1s0nl8p4 flavour physical lc 8 port 4 splittable true lanes 4 > pci/0000:01:00.0/57: type eth netdev enp1s0nl8p5 flavour physical lc 8 port 5 splittable true lanes 4 > pci/0000:01:00.0/58: type eth netdev enp1s0nl8p6 flavour physical lc 8 port 6 splittable true lanes 4 > pci/0000:01:00.0/59: type eth netdev enp1s0nl8p7 flavour physical lc 8 port 7 splittable true lanes 4 > pci/0000:01:00.0/60: type eth netdev enp1s0nl8p8 flavour physical lc 8 port 8 splittable true lanes 4 > pci/0000:01:00.0/61: type eth netdev enp1s0nl8p9 flavour physical lc 8 port 9 splittable true lanes 4 > pci/0000:01:00.0/62: type eth netdev enp1s0nl8p10 flavour physical lc 8 port 10 splittable true lanes 4 > pci/0000:01:00.0/63: type eth netdev enp1s0nl8p11 flavour physical lc 8 port 11 splittable true lanes 4 > pci/0000:01:00.0/64: type eth netdev enp1s0nl8p12 flavour physical lc 8 port 12 splittable true lanes 4 > pci/0000:01:00.0/125: type eth netdev enp1s0nl8p13 flavour physical lc 8 port 13 splittable true lanes 4 > pci/0000:01:00.0/126: type eth netdev enp1s0nl8p14 flavour physical lc 8 port 14 splittable true lanes 4 > pci/0000:01:00.0/127: type eth netdev enp1s0nl8p15 flavour physical lc 8 port 15 splittable true lanes 4 > pci/0000:01:00.0/128: type eth netdev enp1s0nl8p16 flavour physical lc 8 port 16 splittable true lanes 4 > > To uprovision the slot #8: > > $ devlink lc set pci/0000:01:00.0 lc 8 notype > are there any changes from the last RFC? https://lore.kernel.org/netdev/20210122094648.1631078-1-jiri@resnulli.us/
Mon, Apr 18, 2022 at 04:31:30PM CEST, dsahern@gmail.com wrote: >On 4/18/22 12:42 AM, Ido Schimmel wrote: >> Jiri says: >> >> This patchset introduces support for modular switch systems and also >> introduces mlxsw support for NVIDIA Mellanox SN4800 modular switch. >> It contains 8 slots to accommodate line cards - replaceable PHY modules >> which may contain gearboxes. >> Currently supported line card: >> 16X 100GbE (QSFP28) >> Other line cards that are going to be supported: >> 8X 200GbE (QSFP56) >> 4X 400GbE (QSFP-DD) >> There may be other types of line cards added in the future. >> >> To be consistent with the port split configuration (splitter cabels), >> the line card entities are treated in the similar way. The nature of >> a line card is not "a pluggable device", but "a pluggable PHY module". >> >> A concept of "provisioning" is introduced. The user may "provision" >> certain slot with a line card type. Driver then creates all instances >> (devlink ports, netdevices, etc) related to this line card type. It does >> not matter if the line card is plugged-in at the time. User is able to >> configure netdevices, devlink ports, setup port splitters, etc. From the >> perspective of the switch ASIC, all is present and can be configured. >> >> The carrier of netdevices stays down if the line card is not plugged-in. >> Once the line card is inserted and activated, the carrier of >> the related netdevices is then reflecting the physical line state, >> same as for an ordinary fixed port. >> >> Once user does not want to use the line card related instances >> anymore, he can "unprovision" the slot. Driver then removes the >> instances. >> >> Patches 1-4 are extending devlink driver API and UAPI in order to >> register, show, dump, provision and activate the line card. >> Patches 5-17 are implementing the introduced API in mlxsw. >> The last patch adds a selftest for mlxsw line cards. >> >> Example: >> $ devlink port # No ports are listed >> $ devlink lc >> pci/0000:01:00.0: >> lc 1 state unprovisioned >> supported_types: >> 16x100G >> lc 2 state unprovisioned >> supported_types: >> 16x100G >> lc 3 state unprovisioned >> supported_types: >> 16x100G >> lc 4 state unprovisioned >> supported_types: >> 16x100G >> lc 5 state unprovisioned >> supported_types: >> 16x100G >> lc 6 state unprovisioned >> supported_types: >> 16x100G >> lc 7 state unprovisioned >> supported_types: >> 16x100G >> lc 8 state unprovisioned >> supported_types: >> 16x100G >> >> Note that driver exposes list supported line card types. Currently >> there is only one: "16x100G". >> >> To provision the slot #8: >> >> $ devlink lc set pci/0000:01:00.0 lc 8 type 16x100G >> $ devlink lc show pci/0000:01:00.0 lc 8 >> pci/0000:01:00.0: >> lc 8 state active type 16x100G >> supported_types: >> 16x100G >> $ devlink port >> pci/0000:01:00.0/0: type notset flavour cpu port 0 splittable false >> pci/0000:01:00.0/53: type eth netdev enp1s0nl8p1 flavour physical lc 8 port 1 splittable true lanes 4 >> pci/0000:01:00.0/54: type eth netdev enp1s0nl8p2 flavour physical lc 8 port 2 splittable true lanes 4 >> pci/0000:01:00.0/55: type eth netdev enp1s0nl8p3 flavour physical lc 8 port 3 splittable true lanes 4 >> pci/0000:01:00.0/56: type eth netdev enp1s0nl8p4 flavour physical lc 8 port 4 splittable true lanes 4 >> pci/0000:01:00.0/57: type eth netdev enp1s0nl8p5 flavour physical lc 8 port 5 splittable true lanes 4 >> pci/0000:01:00.0/58: type eth netdev enp1s0nl8p6 flavour physical lc 8 port 6 splittable true lanes 4 >> pci/0000:01:00.0/59: type eth netdev enp1s0nl8p7 flavour physical lc 8 port 7 splittable true lanes 4 >> pci/0000:01:00.0/60: type eth netdev enp1s0nl8p8 flavour physical lc 8 port 8 splittable true lanes 4 >> pci/0000:01:00.0/61: type eth netdev enp1s0nl8p9 flavour physical lc 8 port 9 splittable true lanes 4 >> pci/0000:01:00.0/62: type eth netdev enp1s0nl8p10 flavour physical lc 8 port 10 splittable true lanes 4 >> pci/0000:01:00.0/63: type eth netdev enp1s0nl8p11 flavour physical lc 8 port 11 splittable true lanes 4 >> pci/0000:01:00.0/64: type eth netdev enp1s0nl8p12 flavour physical lc 8 port 12 splittable true lanes 4 >> pci/0000:01:00.0/125: type eth netdev enp1s0nl8p13 flavour physical lc 8 port 13 splittable true lanes 4 >> pci/0000:01:00.0/126: type eth netdev enp1s0nl8p14 flavour physical lc 8 port 14 splittable true lanes 4 >> pci/0000:01:00.0/127: type eth netdev enp1s0nl8p15 flavour physical lc 8 port 15 splittable true lanes 4 >> pci/0000:01:00.0/128: type eth netdev enp1s0nl8p16 flavour physical lc 8 port 16 splittable true lanes 4 >> >> To uprovision the slot #8: >> >> $ devlink lc set pci/0000:01:00.0 lc 8 notype >> > >are there any changes from the last RFC? > >https://lore.kernel.org/netdev/20210122094648.1631078-1-jiri@resnulli.us/ Yes, many of them, I din't track them. Mainly, the RFC was backed by netdevsim implementation, this is mlxsw with actual HW underneath.
On Tue, Apr 19, 2022 at 01:55:44PM +0200, Jiri Pirko wrote: > Mon, Apr 18, 2022 at 04:31:30PM CEST, dsahern@gmail.com wrote: > >On 4/18/22 12:42 AM, Ido Schimmel wrote: > >> Jiri says: > >> > >> This patchset introduces support for modular switch systems and also > >> introduces mlxsw support for NVIDIA Mellanox SN4800 modular switch. > >> It contains 8 slots to accommodate line cards - replaceable PHY modules > >> which may contain gearboxes. > >> Currently supported line card: > >> 16X 100GbE (QSFP28) > >> Other line cards that are going to be supported: > >> 8X 200GbE (QSFP56) > >> 4X 400GbE (QSFP-DD) > >> There may be other types of line cards added in the future. > >> > >> To be consistent with the port split configuration (splitter cabels), > >> the line card entities are treated in the similar way. The nature of > >> a line card is not "a pluggable device", but "a pluggable PHY module". > >> > >> A concept of "provisioning" is introduced. The user may "provision" > >> certain slot with a line card type. Driver then creates all instances > >> (devlink ports, netdevices, etc) related to this line card type. It does > >> not matter if the line card is plugged-in at the time. User is able to > >> configure netdevices, devlink ports, setup port splitters, etc. From the > >> perspective of the switch ASIC, all is present and can be configured. > >> > >> The carrier of netdevices stays down if the line card is not plugged-in. > >> Once the line card is inserted and activated, the carrier of > >> the related netdevices is then reflecting the physical line state, > >> same as for an ordinary fixed port. > >> > >> Once user does not want to use the line card related instances > >> anymore, he can "unprovision" the slot. Driver then removes the > >> instances. > >> > >> Patches 1-4 are extending devlink driver API and UAPI in order to > >> register, show, dump, provision and activate the line card. > >> Patches 5-17 are implementing the introduced API in mlxsw. > >> The last patch adds a selftest for mlxsw line cards. > >> > >> Example: > >> $ devlink port # No ports are listed > >> $ devlink lc > >> pci/0000:01:00.0: > >> lc 1 state unprovisioned > >> supported_types: > >> 16x100G > >> lc 2 state unprovisioned > >> supported_types: > >> 16x100G > >> lc 3 state unprovisioned > >> supported_types: > >> 16x100G > >> lc 4 state unprovisioned > >> supported_types: > >> 16x100G > >> lc 5 state unprovisioned > >> supported_types: > >> 16x100G > >> lc 6 state unprovisioned > >> supported_types: > >> 16x100G > >> lc 7 state unprovisioned > >> supported_types: > >> 16x100G > >> lc 8 state unprovisioned > >> supported_types: > >> 16x100G > >> > >> Note that driver exposes list supported line card types. Currently > >> there is only one: "16x100G". > >> > >> To provision the slot #8: > >> > >> $ devlink lc set pci/0000:01:00.0 lc 8 type 16x100G > >> $ devlink lc show pci/0000:01:00.0 lc 8 > >> pci/0000:01:00.0: > >> lc 8 state active type 16x100G > >> supported_types: > >> 16x100G > >> $ devlink port > >> pci/0000:01:00.0/0: type notset flavour cpu port 0 splittable false > >> pci/0000:01:00.0/53: type eth netdev enp1s0nl8p1 flavour physical lc 8 port 1 splittable true lanes 4 > >> pci/0000:01:00.0/54: type eth netdev enp1s0nl8p2 flavour physical lc 8 port 2 splittable true lanes 4 > >> pci/0000:01:00.0/55: type eth netdev enp1s0nl8p3 flavour physical lc 8 port 3 splittable true lanes 4 > >> pci/0000:01:00.0/56: type eth netdev enp1s0nl8p4 flavour physical lc 8 port 4 splittable true lanes 4 > >> pci/0000:01:00.0/57: type eth netdev enp1s0nl8p5 flavour physical lc 8 port 5 splittable true lanes 4 > >> pci/0000:01:00.0/58: type eth netdev enp1s0nl8p6 flavour physical lc 8 port 6 splittable true lanes 4 > >> pci/0000:01:00.0/59: type eth netdev enp1s0nl8p7 flavour physical lc 8 port 7 splittable true lanes 4 > >> pci/0000:01:00.0/60: type eth netdev enp1s0nl8p8 flavour physical lc 8 port 8 splittable true lanes 4 > >> pci/0000:01:00.0/61: type eth netdev enp1s0nl8p9 flavour physical lc 8 port 9 splittable true lanes 4 > >> pci/0000:01:00.0/62: type eth netdev enp1s0nl8p10 flavour physical lc 8 port 10 splittable true lanes 4 > >> pci/0000:01:00.0/63: type eth netdev enp1s0nl8p11 flavour physical lc 8 port 11 splittable true lanes 4 > >> pci/0000:01:00.0/64: type eth netdev enp1s0nl8p12 flavour physical lc 8 port 12 splittable true lanes 4 > >> pci/0000:01:00.0/125: type eth netdev enp1s0nl8p13 flavour physical lc 8 port 13 splittable true lanes 4 > >> pci/0000:01:00.0/126: type eth netdev enp1s0nl8p14 flavour physical lc 8 port 14 splittable true lanes 4 > >> pci/0000:01:00.0/127: type eth netdev enp1s0nl8p15 flavour physical lc 8 port 15 splittable true lanes 4 > >> pci/0000:01:00.0/128: type eth netdev enp1s0nl8p16 flavour physical lc 8 port 16 splittable true lanes 4 > >> > >> To uprovision the slot #8: > >> > >> $ devlink lc set pci/0000:01:00.0 lc 8 notype > >> > > > >are there any changes from the last RFC? > > > >https://lore.kernel.org/netdev/20210122094648.1631078-1-jiri@resnulli.us/ > > Yes, many of them, I din't track them. Mainly, the RFC was backed by > netdevsim implementation, this is mlxsw with actual HW underneath. I don't think David was asking about the backing implementation, but about the interface itself. I don't remember any changes in this aspect during the internal review and I also compared the uAPI files and they seem to be the same. The only change that I do remember is making the APIs idempotent. Previously, this would fail: # devlink lc set pci/0000:01:00.0 lc 8 notype # devlink lc set pci/0000:01:00.0 lc 8 notype