| Message ID | 20240628133517.8591-1-gakula@marvell.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Introduce RVU representors | expand |
On Fri, 28 Jun 2024 19:05:07 +0530 Geetha sowjanya wrote: > This series adds representor support for each rvu devices. > When switchdev mode is enabled, representor netdev is registered > for each rvu device. In implementation of representor model, > one NIX HW LF with multiple SQ and RQ is reserved, where each > RQ and SQ of the LF are mapped to a representor. A loopback channel > is reserved to support packet path between representors and VFs. > CN10K silicon supports 2 types of MACs, RPM and SDP. This > patch set adds representor support for both RPM and SDP MAC > interfaces. > > - Patch 1: Refactors and exports the shared service functions. > - Patch 2: Implements basic representor driver. > - Patch 3: Add devlink support to create representor netdevs that > can be used to manage VFs. > - Patch 4: Implements basec netdev_ndo_ops. > - Patch 5: Installs tcam rules to route packets between representor and > VFs. > - Patch 6: Enables fetching VF stats via representor interface > - Patch 7: Adds support to sync link state between representors and VFs . > - Patch 8: Enables configuring VF MTU via representor netdevs. > - Patch 9: Add representors for sdp MAC. > - Patch 10: Add devlink port support. > > Command to create VF representor > #devlink dev eswitch set pci/0002:1c:00.0 mode switchdev > VF representors are created for each VF when switch mode is set switchdev on representor PCI device > # devlink dev eswitch set pci/0002:1c:00.0 mode switchdev > # ip link show > 25: r0p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 > link/ether 32:0f:0f:f0:60:f1 brd ff:ff:ff:ff:ff:ff > 26: r1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 > link/ether 3e:5d:9a:4d:e7:7b brd ff:ff:ff:ff:ff:ff > > #devlink dev > pci/0002:01:00.0 > pci/0002:02:00.0 > pci/0002:03:00.0 > pci/0002:04:00.0 > pci/0002:05:00.0 > pci/0002:06:00.0 > pci/0002:07:00.0 > > ~# devlink port > pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 pfnum 1 vfnum 0 external false splittable false > pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 pfnum 1 vfnum 1 external false splittable false > pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 pfnum 1 vfnum 2 external false splittable false > pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 pfnum 1 vfnum 3 external false splittable false Please document the state before and after switching modes, and record this information in Documentation/networking/device_drivers/ethernet/marvell/octeontx2.rst In the commands above you seem to operate on a pci/0002:1c:00.0 devlink instance, and yet devlink dev does not list it. You have 2 netdevs, 8 devlinks, 4 ports, and no PF instance. It does not add up..
>-----Original Message----- >From: Jakub Kicinski <kuba@kernel.org> >Sent: Tuesday, July 2, 2024 8:42 AM >To: Geethasowjanya Akula <gakula@marvell.com> >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; >davem@davemloft.net; pabeni@redhat.com; edumazet@google.com; Sunil >Kovvuri Goutham <sgoutham@marvell.com>; Subbaraya Sundeep Bhatta ><sbhatta@marvell.com>; Hariprasad Kelam <hkelam@marvell.com> >Subject: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU >representors > >On Fri, 28 Jun 2024 19:05:07 +0530 Geetha sowjanya wrote: >> This series adds representor support for each rvu devices. >> When switchdev mode is enabled, representor netdev is registered for >> each rvu device. In implementation of representor model, one NIX HW LF >> with multiple SQ and RQ is reserved, where each RQ and SQ of the LF >> are mapped to a representor. A loopback channel is reserved to support >> packet path between representors and VFs. >> CN10K silicon supports 2 types of MACs, RPM and SDP. This patch set >> adds representor support for both RPM and SDP MAC interfaces. >> >> - Patch 1: Refactors and exports the shared service functions. >> - Patch 2: Implements basic representor driver. >> - Patch 3: Add devlink support to create representor netdevs that >> can be used to manage VFs. >> - Patch 4: Implements basec netdev_ndo_ops. >> - Patch 5: Installs tcam rules to route packets between representor and >> VFs. >> - Patch 6: Enables fetching VF stats via representor interface >> - Patch 7: Adds support to sync link state between representors and VFs . >> - Patch 8: Enables configuring VF MTU via representor netdevs. >> - Patch 9: Add representors for sdp MAC. >> - Patch 10: Add devlink port support. >> >> Command to create VF representor >> #devlink dev eswitch set pci/0002:1c:00.0 mode switchdev VF >> representors are created for each VF when switch mode is set switchdev >> on representor PCI device # devlink dev eswitch set pci/0002:1c:00.0 >> mode switchdev # ip link show >> 25: r0p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode DEFAULT group default qlen 1000 >> link/ether 32:0f:0f:f0:60:f1 brd ff:ff:ff:ff:ff:ff >> 26: r1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode DEFAULT group default qlen 1000 >> link/ether 3e:5d:9a:4d:e7:7b brd ff:ff:ff:ff:ff:ff >> >> #devlink dev >> pci/0002:01:00.0 >> pci/0002:02:00.0 >> pci/0002:03:00.0 >> pci/0002:04:00.0 >> pci/0002:05:00.0 >> pci/0002:06:00.0 >> pci/0002:07:00.0 >> >> ~# devlink port >> pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 >> pfnum 1 vfnum 0 external false splittable false >> pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 >> pfnum 1 vfnum 1 external false splittable false >> pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 >> pfnum 1 vfnum 2 external false splittable false >> pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 >> pfnum 1 vfnum 3 external false splittable false > >Please document the state before and after switching modes, and record this >information in >Documentation/networking/device_drivers/ethernet/marvell/octeontx2.rst Ok. Will next version with documentation. > >In the commands above you seem to operate on a pci/0002:1c:00.0 devlink >instance, and yet devlink dev does not list it. You have 2 netdevs, >8 devlinks, 4 ports, and no PF instance. It does not add up.. The log was large. When I cut short actual device was missing. Will provide complete log in the documentation.
On Fri, Jun 28, 2024 at 07:05:07PM +0530, Geetha sowjanya wrote: > This series adds representor support for each rvu devices. > When switchdev mode is enabled, representor netdev is registered > for each rvu device. In implementation of representor model, > one NIX HW LF with multiple SQ and RQ is reserved, where each > RQ and SQ of the LF are mapped to a representor. A loopback channel > is reserved to support packet path between representors and VFs. > CN10K silicon supports 2 types of MACs, RPM and SDP. This > patch set adds representor support for both RPM and SDP MAC > interfaces. > > - Patch 1: Refactors and exports the shared service functions. > - Patch 2: Implements basic representor driver. > - Patch 3: Add devlink support to create representor netdevs that > can be used to manage VFs. > - Patch 4: Implements basec netdev_ndo_ops. > - Patch 5: Installs tcam rules to route packets between representor and > VFs. > - Patch 6: Enables fetching VF stats via representor interface > - Patch 7: Adds support to sync link state between representors and VFs . > - Patch 8: Enables configuring VF MTU via representor netdevs. > - Patch 9: Add representors for sdp MAC. > - Patch 10: Add devlink port support. > > Command to create VF representor > #devlink dev eswitch set pci/0002:1c:00.0 mode switchdev > VF representors are created for each VF when switch mode is set switchdev on representor PCI device Does it mean that VFs needs to be created before going to switchdev mode? (in legacy mode). Keep in mind that in both mellanox and ice driver assume that VFs are created after chaning mode to switchdev (mode can't be changed if VFs). Different order can be problematic. For example (AFAIK) kubernetes scripts for switchdev assume that first is switching to switchdev and VFs creation is done after that. Thanks, Michal > # devlink dev eswitch set pci/0002:1c:00.0 mode switchdev > # ip link show > 25: r0p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 > link/ether 32:0f:0f:f0:60:f1 brd ff:ff:ff:ff:ff:ff > 26: r1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 > link/ether 3e:5d:9a:4d:e7:7b brd ff:ff:ff:ff:ff:ff > > #devlink dev > pci/0002:01:00.0 > pci/0002:02:00.0 > pci/0002:03:00.0 > pci/0002:04:00.0 > pci/0002:05:00.0 > pci/0002:06:00.0 > pci/0002:07:00.0 > > ~# devlink port > pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 pfnum 1 vfnum 0 external false splittable false > pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 pfnum 1 vfnum 1 external false splittable false > pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 pfnum 1 vfnum 2 external false splittable false > pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 pfnum 1 vfnum 3 external false splittable false > > ----------- > v1-v2: > -Fixed build warnings. > -Address review comments provided by "Kalesh Anakkur Purayil". > > v2-v3: > - Used extack for error messages. > - As suggested reworked commit messages. > - Fixed sparse warning. > > v3-v4: > - Patch 2 & 3: Fixed coccinelle reported warnings. > - Patch 10: Added devlink port support. > > v4-v5: > - Patch 3: Removed devm_* usage in rvu_rep_create() > - Patch 3: Fixed build warnings. > > v5-v6: > - Addressed review comments provided by "Simon Horman". > - Added review tag. > > v6-v7: > - Rebased on top net-next branch. > > Geetha sowjanya (10): > octeontx2-pf: Refactoring RVU driver > octeontx2-pf: RVU representor driver > octeontx2-pf: Create representor netdev > octeontx2-pf: Add basic net_device_ops > octeontx2-af: Add packet path between representor and VF > octeontx2-pf: Get VF stats via representor > octeontx2-pf: Add support to sync link state between representor and > VFs > octeontx2-pf: Configure VF mtu via representor > octeontx2-pf: Add representors for sdp MAC > octeontx2-pf: Add devlink port support > > .../net/ethernet/marvell/octeontx2/Kconfig | 8 + > .../ethernet/marvell/octeontx2/af/Makefile | 3 +- > .../ethernet/marvell/octeontx2/af/common.h | 2 + > .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ > .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + > .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + > .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- > .../marvell/octeontx2/af/rvu_debugfs.c | 27 - > .../marvell/octeontx2/af/rvu_devlink.c | 6 + > .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 ++- > .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + > .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + > .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 ++++++++++++ > .../marvell/octeontx2/af/rvu_struct.h | 26 + > .../marvell/octeontx2/af/rvu_switch.c | 20 +- > .../ethernet/marvell/octeontx2/nic/Makefile | 2 + > .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- > .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- > .../marvell/octeontx2/nic/otx2_common.c | 56 +- > .../marvell/octeontx2/nic/otx2_common.h | 84 ++- > .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ > .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- > .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- > .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- > .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- > .../net/ethernet/marvell/octeontx2/nic/rep.c | 684 ++++++++++++++++++ > .../net/ethernet/marvell/octeontx2/nic/rep.h | 53 ++ > 27 files changed, 1834 insertions(+), 227 deletions(-) > create mode 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c > create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.c > create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.h > > -- > 2.25.1 >
>-----Original Message----- >From: Michal Swiatkowski <michal.swiatkowski@linux.intel.com> >Sent: Wednesday, July 3, 2024 4:14 PM >To: Geethasowjanya Akula <gakula@marvell.com> >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; kuba@kernel.org; >davem@davemloft.net; pabeni@redhat.com; edumazet@google.com; Sunil >Kovvuri Goutham <sgoutham@marvell.com>; Subbaraya Sundeep Bhatta ><sbhatta@marvell.com>; Hariprasad Kelam <hkelam@marvell.com> >Subject: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU representors >On Fri, Jun 28, 2024 at 07:05:07PM +0530, Geetha sowjanya wrote: >> This series adds representor support for each rvu devices. >> When switchdev mode is enabled, representor netdev is registered for >> each rvu device. In implementation of representor model, one NIX HW LF >> with multiple SQ and RQ is reserved, where each RQ and SQ of the LF >> are mapped to a representor. A loopback channel is reserved to support >> packet path between representors and VFs. >> CN10K silicon supports 2 types of MACs, RPM and SDP. This patch set >> adds representor support for both RPM and SDP MAC interfaces. >> >> - Patch 1: Refactors and exports the shared service functions. >> - Patch 2: Implements basic representor driver. >> - Patch 3: Add devlink support to create representor netdevs that >> can be used to manage VFs. >> - Patch 4: Implements basec netdev_ndo_ops. >> - Patch 5: Installs tcam rules to route packets between representor and >> VFs. >> - Patch 6: Enables fetching VF stats via representor interface >> - Patch 7: Adds support to sync link state between representors and VFs . >> - Patch 8: Enables configuring VF MTU via representor netdevs. >> - Patch 9: Add representors for sdp MAC. >> - Patch 10: Add devlink port support. >> >> Command to create VF representor >> #devlink dev eswitch set pci/0002:1c:00.0 mode switchdev VF >> representors are created for each VF when switch mode is set switchdev >> on representor PCI device > >Does it mean that VFs needs to be created before going to switchdev mode? (in >legacy mode). Keep in mind that in both mellanox and ice driver assume that >VFs are created after chaning mode to switchdev (mode can't be changed if >VFs). No. RVU representor driver implementation is similar to mellanox and ice drivers. It assumes that VF gets created only after switchdev mode is enabled. Sorry, if above commit description is confusing. Will rewrite it. > >Different order can be problematic. For example (AFAIK) kubernetes scripts for >switchdev assume that first is switching to switchdev and VFs creation is done >after that. > >Thanks, >Michal > >> # devlink dev eswitch set pci/0002:1c:00.0 mode switchdev # ip link >> show >> 25: r0p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode >DEFAULT group default qlen 1000 >> link/ether 32:0f:0f:f0:60:f1 brd ff:ff:ff:ff:ff:ff >> 26: r1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode >DEFAULT group default qlen 1000 >> link/ether 3e:5d:9a:4d:e7:7b brd ff:ff:ff:ff:ff:ff >> >> #devlink dev >> pci/0002:01:00.0 >> pci/0002:02:00.0 >> pci/0002:03:00.0 >> pci/0002:04:00.0 >> pci/0002:05:00.0 >> pci/0002:06:00.0 >> pci/0002:07:00.0 >> >> ~# devlink port >> pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 >> pfnum 1 vfnum 0 external false splittable false >> pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 >> pfnum 1 vfnum 1 external false splittable false >> pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 >> pfnum 1 vfnum 2 external false splittable false >> pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 >> pfnum 1 vfnum 3 external false splittable false >> >> ----------- >> v1-v2: >> -Fixed build warnings. >> -Address review comments provided by "Kalesh Anakkur Purayil". >> >> v2-v3: >> - Used extack for error messages. >> - As suggested reworked commit messages. >> - Fixed sparse warning. >> >> v3-v4: >> - Patch 2 & 3: Fixed coccinelle reported warnings. >> - Patch 10: Added devlink port support. >> >> v4-v5: >> - Patch 3: Removed devm_* usage in rvu_rep_create() >> - Patch 3: Fixed build warnings. >> >> v5-v6: >> - Addressed review comments provided by "Simon Horman". >> - Added review tag. >> >> v6-v7: >> - Rebased on top net-next branch. >> >> Geetha sowjanya (10): >> octeontx2-pf: Refactoring RVU driver >> octeontx2-pf: RVU representor driver >> octeontx2-pf: Create representor netdev >> octeontx2-pf: Add basic net_device_ops >> octeontx2-af: Add packet path between representor and VF >> octeontx2-pf: Get VF stats via representor >> octeontx2-pf: Add support to sync link state between representor and >> VFs >> octeontx2-pf: Configure VF mtu via representor >> octeontx2-pf: Add representors for sdp MAC >> octeontx2-pf: Add devlink port support >> >> .../net/ethernet/marvell/octeontx2/Kconfig | 8 + >> .../ethernet/marvell/octeontx2/af/Makefile | 3 +- >> .../ethernet/marvell/octeontx2/af/common.h | 2 + >> .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ >> .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + >> .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + >> .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- >> .../marvell/octeontx2/af/rvu_debugfs.c | 27 - >> .../marvell/octeontx2/af/rvu_devlink.c | 6 + >> .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 ++- >> .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + >> .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + >> .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 ++++++++++++ >> .../marvell/octeontx2/af/rvu_struct.h | 26 + >> .../marvell/octeontx2/af/rvu_switch.c | 20 +- >> .../ethernet/marvell/octeontx2/nic/Makefile | 2 + >> .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- >> .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- >> .../marvell/octeontx2/nic/otx2_common.c | 56 +- >> .../marvell/octeontx2/nic/otx2_common.h | 84 ++- >> .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ >> .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- >> .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- >> .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- >> .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- >> .../net/ethernet/marvell/octeontx2/nic/rep.c | 684 ++++++++++++++++++ >> .../net/ethernet/marvell/octeontx2/nic/rep.h | 53 ++ >> 27 files changed, 1834 insertions(+), 227 deletions(-) create mode >> 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c >> create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.c >> create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.h >> >> -- >> 2.25.1 >>
On Wed, Jul 03, 2024 at 02:34:03PM +0000, Geethasowjanya Akula wrote: > > > >-----Original Message----- > >From: Michal Swiatkowski <michal.swiatkowski@linux.intel.com> > >Sent: Wednesday, July 3, 2024 4:14 PM > >To: Geethasowjanya Akula <gakula@marvell.com> > >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; kuba@kernel.org; > >davem@davemloft.net; pabeni@redhat.com; edumazet@google.com; Sunil > >Kovvuri Goutham <sgoutham@marvell.com>; Subbaraya Sundeep Bhatta > ><sbhatta@marvell.com>; Hariprasad Kelam <hkelam@marvell.com> > >Subject: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU representors > >On Fri, Jun 28, 2024 at 07:05:07PM +0530, Geetha sowjanya wrote: > >> This series adds representor support for each rvu devices. > >> When switchdev mode is enabled, representor netdev is registered for > >> each rvu device. In implementation of representor model, one NIX HW LF > >> with multiple SQ and RQ is reserved, where each RQ and SQ of the LF > >> are mapped to a representor. A loopback channel is reserved to support > >> packet path between representors and VFs. > >> CN10K silicon supports 2 types of MACs, RPM and SDP. This patch set > >> adds representor support for both RPM and SDP MAC interfaces. > >> > >> - Patch 1: Refactors and exports the shared service functions. > >> - Patch 2: Implements basic representor driver. > >> - Patch 3: Add devlink support to create representor netdevs that > >> can be used to manage VFs. > >> - Patch 4: Implements basec netdev_ndo_ops. > >> - Patch 5: Installs tcam rules to route packets between representor and > >> VFs. > >> - Patch 6: Enables fetching VF stats via representor interface > >> - Patch 7: Adds support to sync link state between representors and VFs . > >> - Patch 8: Enables configuring VF MTU via representor netdevs. > >> - Patch 9: Add representors for sdp MAC. > >> - Patch 10: Add devlink port support. > >> > >> Command to create VF representor > >> #devlink dev eswitch set pci/0002:1c:00.0 mode switchdev VF > >> representors are created for each VF when switch mode is set switchdev > >> on representor PCI device > > > >Does it mean that VFs needs to be created before going to switchdev mode? (in > >legacy mode). Keep in mind that in both mellanox and ice driver assume that > >VFs are created after chaning mode to switchdev (mode can't be changed if > >VFs). > No. RVU representor driver implementation is similar to mellanox and ice drivers. > It assumes that VF gets created only after switchdev mode is enabled. > Sorry, if above commit description is confusing. Will rewrite it. > Ok, but why the rvu_rep_create() is called in switching mode to switchdev function? In this function you are creating netdevs, only for PF representor? It looks like it doesn't called from other context in this patchset, so where the port representor netdevs for VFs are created? Thanks, Michal > > > >Different order can be problematic. For example (AFAIK) kubernetes scripts for > >switchdev assume that first is switching to switchdev and VFs creation is done > >after that. > > > >Thanks, > >Michal > > > >> # devlink dev eswitch set pci/0002:1c:00.0 mode switchdev # ip link > >> show > >> 25: r0p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode > >DEFAULT group default qlen 1000 > >> link/ether 32:0f:0f:f0:60:f1 brd ff:ff:ff:ff:ff:ff > >> 26: r1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode > >DEFAULT group default qlen 1000 > >> link/ether 3e:5d:9a:4d:e7:7b brd ff:ff:ff:ff:ff:ff > >> > >> #devlink dev > >> pci/0002:01:00.0 > >> pci/0002:02:00.0 > >> pci/0002:03:00.0 > >> pci/0002:04:00.0 > >> pci/0002:05:00.0 > >> pci/0002:06:00.0 > >> pci/0002:07:00.0 > >> > >> ~# devlink port > >> pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 > >> pfnum 1 vfnum 0 external false splittable false > >> pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 > >> pfnum 1 vfnum 1 external false splittable false > >> pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 > >> pfnum 1 vfnum 2 external false splittable false > >> pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 > >> pfnum 1 vfnum 3 external false splittable false > >> > >> ----------- > >> v1-v2: > >> -Fixed build warnings. > >> -Address review comments provided by "Kalesh Anakkur Purayil". > >> > >> v2-v3: > >> - Used extack for error messages. > >> - As suggested reworked commit messages. > >> - Fixed sparse warning. > >> > >> v3-v4: > >> - Patch 2 & 3: Fixed coccinelle reported warnings. > >> - Patch 10: Added devlink port support. > >> > >> v4-v5: > >> - Patch 3: Removed devm_* usage in rvu_rep_create() > >> - Patch 3: Fixed build warnings. > >> > >> v5-v6: > >> - Addressed review comments provided by "Simon Horman". > >> - Added review tag. > >> > >> v6-v7: > >> - Rebased on top net-next branch. > >> > >> Geetha sowjanya (10): > >> octeontx2-pf: Refactoring RVU driver > >> octeontx2-pf: RVU representor driver > >> octeontx2-pf: Create representor netdev > >> octeontx2-pf: Add basic net_device_ops > >> octeontx2-af: Add packet path between representor and VF > >> octeontx2-pf: Get VF stats via representor > >> octeontx2-pf: Add support to sync link state between representor and > >> VFs > >> octeontx2-pf: Configure VF mtu via representor > >> octeontx2-pf: Add representors for sdp MAC > >> octeontx2-pf: Add devlink port support > >> > >> .../net/ethernet/marvell/octeontx2/Kconfig | 8 + > >> .../ethernet/marvell/octeontx2/af/Makefile | 3 +- > >> .../ethernet/marvell/octeontx2/af/common.h | 2 + > >> .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ > >> .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + > >> .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + > >> .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- > >> .../marvell/octeontx2/af/rvu_debugfs.c | 27 - > >> .../marvell/octeontx2/af/rvu_devlink.c | 6 + > >> .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 ++- > >> .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + > >> .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + > >> .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 ++++++++++++ > >> .../marvell/octeontx2/af/rvu_struct.h | 26 + > >> .../marvell/octeontx2/af/rvu_switch.c | 20 +- > >> .../ethernet/marvell/octeontx2/nic/Makefile | 2 + > >> .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- > >> .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- > >> .../marvell/octeontx2/nic/otx2_common.c | 56 +- > >> .../marvell/octeontx2/nic/otx2_common.h | 84 ++- > >> .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ > >> .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- > >> .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- > >> .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- > >> .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- > >> .../net/ethernet/marvell/octeontx2/nic/rep.c | 684 ++++++++++++++++++ > >> .../net/ethernet/marvell/octeontx2/nic/rep.h | 53 ++ > >> 27 files changed, 1834 insertions(+), 227 deletions(-) create mode > >> 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c > >> create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.c > >> create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.h > >> > >> -- > >> 2.25.1 > >>
>-----Original Message----- >From: Michal Swiatkowski <michal.swiatkowski@linux.intel.com> >Sent: Thursday, July 4, 2024 12:45 PM >To: Geethasowjanya Akula <gakula@marvell.com> >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; kuba@kernel.org; >davem@davemloft.net; pabeni@redhat.com; edumazet@google.com; Sunil >Kovvuri Goutham <sgoutham@marvell.com>; Subbaraya Sundeep Bhatta ><sbhatta@marvell.com>; Hariprasad Kelam <hkelam@marvell.com> >Subject: Re: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU >representors > >On Wed, Jul 03, 2024 at 02: 34: 03PM +0000, Geethasowjanya Akula wrote: > > >> >-----Original Message----- > >From: Michal Swiatkowski ><michal. swiatkowski@ linux. intel. com> > >Sent: Wednesday, July 3, 2024 4: 14 >PM >On Wed, Jul 03, 2024 at 02:34:03PM +0000, Geethasowjanya Akula wrote: >> >> >> >-----Original Message----- >> >From: Michal Swiatkowski <michal.swiatkowski@linux.intel.com> >> >Sent: Wednesday, July 3, 2024 4:14 PM >> >To: Geethasowjanya Akula <gakula@marvell.com> >> >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; >> >kuba@kernel.org; davem@davemloft.net; pabeni@redhat.com; >> >edumazet@google.com; Sunil Kovvuri Goutham ><sgoutham@marvell.com>; >> >Subbaraya Sundeep Bhatta <sbhatta@marvell.com>; Hariprasad Kelam >> ><hkelam@marvell.com> >> >Subject: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU >> >representors On Fri, Jun 28, 2024 at 07:05:07PM +0530, Geetha sowjanya >wrote: >> >> This series adds representor support for each rvu devices. >> >> When switchdev mode is enabled, representor netdev is registered >> >> for each rvu device. In implementation of representor model, one >> >> NIX HW LF with multiple SQ and RQ is reserved, where each RQ and SQ >> >> of the LF are mapped to a representor. A loopback channel is >> >> reserved to support packet path between representors and VFs. >> >> CN10K silicon supports 2 types of MACs, RPM and SDP. This patch set >> >> adds representor support for both RPM and SDP MAC interfaces. >> >> >> >> - Patch 1: Refactors and exports the shared service functions. >> >> - Patch 2: Implements basic representor driver. >> >> - Patch 3: Add devlink support to create representor netdevs that >> >> can be used to manage VFs. >> >> - Patch 4: Implements basec netdev_ndo_ops. >> >> - Patch 5: Installs tcam rules to route packets between representor and >> >> VFs. >> >> - Patch 6: Enables fetching VF stats via representor interface >> >> - Patch 7: Adds support to sync link state between representors and VFs . >> >> - Patch 8: Enables configuring VF MTU via representor netdevs. >> >> - Patch 9: Add representors for sdp MAC. >> >> - Patch 10: Add devlink port support. >> >> >> >> Command to create VF representor >> >> #devlink dev eswitch set pci/0002:1c:00.0 mode switchdev VF >> >> representors are created for each VF when switch mode is set >> >> switchdev on representor PCI device >> > >> >Does it mean that VFs needs to be created before going to switchdev >> >mode? (in legacy mode). Keep in mind that in both mellanox and ice >> >driver assume that VFs are created after chaning mode to switchdev >> >(mode can't be changed if VFs). >> No. RVU representor driver implementation is similar to mellanox and ice >drivers. >> It assumes that VF gets created only after switchdev mode is enabled. >> Sorry, if above commit description is confusing. Will rewrite it. >> > >Ok, but why the rvu_rep_create() is called in switching mode to switchdev >function? In this function you are creating netdevs, only for PF representor? It >looks like it doesn't called from other context in this patchset, so where the >port representor netdevs for VFs are created? > RVU representors for PF/VFs are created when switchdev mode is set, similar to the bnxt and nfp drivers. rvu_rep_create() will create representors based on rep_cnt (which include both PFs and VFs count) Thanks, Geetha. >Thanks, >Michal > >> > >> >Different order can be problematic. For example (AFAIK) kubernetes >> >scripts for switchdev assume that first is switching to switchdev and >> >VFs creation is done after that. >> > >> >Thanks, >> >Michal >> > >> >> # devlink dev eswitch set pci/0002:1c:00.0 mode switchdev # ip >> >> link show >> >> 25: r0p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode >> >DEFAULT group default qlen 1000 >> >> link/ether 32:0f:0f:f0:60:f1 brd ff:ff:ff:ff:ff:ff >> >> 26: r1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >mode >> >DEFAULT group default qlen 1000 >> >> link/ether 3e:5d:9a:4d:e7:7b brd ff:ff:ff:ff:ff:ff >> >> >> >> #devlink dev >> >> pci/0002:01:00.0 >> >> pci/0002:02:00.0 >> >> pci/0002:03:00.0 >> >> pci/0002:04:00.0 >> >> pci/0002:05:00.0 >> >> pci/0002:06:00.0 >> >> pci/0002:07:00.0 >> >> >> >> ~# devlink port >> >> pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller >> >> 0 pfnum 1 vfnum 0 external false splittable false >> >> pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller >> >> 0 pfnum 1 vfnum 1 external false splittable false >> >> pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller >> >> 0 pfnum 1 vfnum 2 external false splittable false >> >> pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller >> >> 0 pfnum 1 vfnum 3 external false splittable false >> >> >> >> ----------- >> >> v1-v2: >> >> -Fixed build warnings. >> >> -Address review comments provided by "Kalesh Anakkur Purayil". >> >> >> >> v2-v3: >> >> - Used extack for error messages. >> >> - As suggested reworked commit messages. >> >> - Fixed sparse warning. >> >> >> >> v3-v4: >> >> - Patch 2 & 3: Fixed coccinelle reported warnings. >> >> - Patch 10: Added devlink port support. >> >> >> >> v4-v5: >> >> - Patch 3: Removed devm_* usage in rvu_rep_create() >> >> - Patch 3: Fixed build warnings. >> >> >> >> v5-v6: >> >> - Addressed review comments provided by "Simon Horman". >> >> - Added review tag. >> >> >> >> v6-v7: >> >> - Rebased on top net-next branch. >> >> >> >> Geetha sowjanya (10): >> >> octeontx2-pf: Refactoring RVU driver >> >> octeontx2-pf: RVU representor driver >> >> octeontx2-pf: Create representor netdev >> >> octeontx2-pf: Add basic net_device_ops >> >> octeontx2-af: Add packet path between representor and VF >> >> octeontx2-pf: Get VF stats via representor >> >> octeontx2-pf: Add support to sync link state between representor and >> >> VFs >> >> octeontx2-pf: Configure VF mtu via representor >> >> octeontx2-pf: Add representors for sdp MAC >> >> octeontx2-pf: Add devlink port support >> >> >> >> .../net/ethernet/marvell/octeontx2/Kconfig | 8 + >> >> .../ethernet/marvell/octeontx2/af/Makefile | 3 +- >> >> .../ethernet/marvell/octeontx2/af/common.h | 2 + >> >> .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ >> >> .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + >> >> .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + >> >> .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- >> >> .../marvell/octeontx2/af/rvu_debugfs.c | 27 - >> >> .../marvell/octeontx2/af/rvu_devlink.c | 6 + >> >> .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 ++- >> >> .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + >> >> .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + >> >> .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 ++++++++++++ >> >> .../marvell/octeontx2/af/rvu_struct.h | 26 + >> >> .../marvell/octeontx2/af/rvu_switch.c | 20 +- >> >> .../ethernet/marvell/octeontx2/nic/Makefile | 2 + >> >> .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- >> >> .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- >> >> .../marvell/octeontx2/nic/otx2_common.c | 56 +- >> >> .../marvell/octeontx2/nic/otx2_common.h | 84 ++- >> >> .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ >> >> .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- >> >> .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- >> >> .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- >> >> .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- >> >> .../net/ethernet/marvell/octeontx2/nic/rep.c | 684 >> >> ++++++++++++++++++ .../net/ethernet/marvell/octeontx2/nic/rep.h | >> >> 53 ++ >> >> 27 files changed, 1834 insertions(+), 227 deletions(-) create >> >> mode >> >> 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c >> >> create mode 100644 >> >> drivers/net/ethernet/marvell/octeontx2/nic/rep.c >> >> create mode 100644 >> >> drivers/net/ethernet/marvell/octeontx2/nic/rep.h >> >> >> >> -- >> >> 2.25.1 >> >>
On Thu, Jul 04, 2024 at 01:48:23PM +0000, Geethasowjanya Akula wrote: > > > >-----Original Message----- > >From: Michal Swiatkowski <michal.swiatkowski@linux.intel.com> > >Sent: Thursday, July 4, 2024 12:45 PM > >To: Geethasowjanya Akula <gakula@marvell.com> > >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; kuba@kernel.org; > >davem@davemloft.net; pabeni@redhat.com; edumazet@google.com; Sunil > >Kovvuri Goutham <sgoutham@marvell.com>; Subbaraya Sundeep Bhatta > ><sbhatta@marvell.com>; Hariprasad Kelam <hkelam@marvell.com> > >Subject: Re: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU > >representors > > > >On Wed, Jul 03, 2024 at 02: 34: 03PM +0000, Geethasowjanya Akula wrote: > > > >> >-----Original Message----- > >From: Michal Swiatkowski > ><michal. swiatkowski@ linux. intel. com> > >Sent: Wednesday, July 3, 2024 4: 14 > >PM > >On Wed, Jul 03, 2024 at 02:34:03PM +0000, Geethasowjanya Akula wrote: > >> > >> > >> >-----Original Message----- > >> >From: Michal Swiatkowski <michal.swiatkowski@linux.intel.com> > >> >Sent: Wednesday, July 3, 2024 4:14 PM > >> >To: Geethasowjanya Akula <gakula@marvell.com> > >> >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; > >> >kuba@kernel.org; davem@davemloft.net; pabeni@redhat.com; > >> >edumazet@google.com; Sunil Kovvuri Goutham > ><sgoutham@marvell.com>; > >> >Subbaraya Sundeep Bhatta <sbhatta@marvell.com>; Hariprasad Kelam > >> ><hkelam@marvell.com> > >> >Subject: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU > >> >representors On Fri, Jun 28, 2024 at 07:05:07PM +0530, Geetha sowjanya > >wrote: > >> >> This series adds representor support for each rvu devices. > >> >> When switchdev mode is enabled, representor netdev is registered > >> >> for each rvu device. In implementation of representor model, one > >> >> NIX HW LF with multiple SQ and RQ is reserved, where each RQ and SQ > >> >> of the LF are mapped to a representor. A loopback channel is > >> >> reserved to support packet path between representors and VFs. > >> >> CN10K silicon supports 2 types of MACs, RPM and SDP. This patch set > >> >> adds representor support for both RPM and SDP MAC interfaces. > >> >> > >> >> - Patch 1: Refactors and exports the shared service functions. > >> >> - Patch 2: Implements basic representor driver. > >> >> - Patch 3: Add devlink support to create representor netdevs that > >> >> can be used to manage VFs. > >> >> - Patch 4: Implements basec netdev_ndo_ops. > >> >> - Patch 5: Installs tcam rules to route packets between representor and > >> >> VFs. > >> >> - Patch 6: Enables fetching VF stats via representor interface > >> >> - Patch 7: Adds support to sync link state between representors and VFs . > >> >> - Patch 8: Enables configuring VF MTU via representor netdevs. > >> >> - Patch 9: Add representors for sdp MAC. > >> >> - Patch 10: Add devlink port support. > >> >> > >> >> Command to create VF representor > >> >> #devlink dev eswitch set pci/0002:1c:00.0 mode switchdev VF > >> >> representors are created for each VF when switch mode is set > >> >> switchdev on representor PCI device > >> > > >> >Does it mean that VFs needs to be created before going to switchdev > >> >mode? (in legacy mode). Keep in mind that in both mellanox and ice > >> >driver assume that VFs are created after chaning mode to switchdev > >> >(mode can't be changed if VFs). > >> No. RVU representor driver implementation is similar to mellanox and ice > >drivers. > >> It assumes that VF gets created only after switchdev mode is enabled. > >> Sorry, if above commit description is confusing. Will rewrite it. > >> > > > >Ok, but why the rvu_rep_create() is called in switching mode to switchdev > >function? In this function you are creating netdevs, only for PF representor? It > >looks like it doesn't called from other context in this patchset, so where the > >port representor netdevs for VFs are created? > > > RVU representors for PF/VFs are created when switchdev mode is set, similar to the bnxt and nfp drivers. > rvu_rep_create() will create representors based on rep_cnt (which include both PFs and VFs count) > Sorry, I don't understand now. You wrote in previous message that VFs should be created after switchdev mode is enabled. Now you are writting that they are created based on rep_cnt (so assuming VFs have been created before). What is the correct order? # echo 1 x > sriov_numvfs # devlink dev eswitch set pci/0000:ca:00.0 mode switchdev or # devlink dev eswitch set pci/0000:ca:00.0 mode switchdev # echo 1 x > sriov_numvfs or you can create VFs before and after? From looking at the drivers code it looks like driver bnxt supports that. I am not familiar with nfp and bnxt driver but based on code in current kernel version it looks like nfp creates representor when switching mode only for physical and PF types, VF representor type is created when the VF is created. Bnxt support VF representor creation before and after switching mode. In ice port representors are created during VFs creation only. I am not against your solution, only pointing that it can lead to some problems. Thanks, Michal > Thanks, > Geetha. > > >Thanks, > >Michal > > > >> > > >> >Different order can be problematic. For example (AFAIK) kubernetes > >> >scripts for switchdev assume that first is switching to switchdev and > >> >VFs creation is done after that. > >> > > >> >Thanks, > >> >Michal > >> > > >> >> # devlink dev eswitch set pci/0002:1c:00.0 mode switchdev # ip > >> >> link show > >> >> 25: r0p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > >mode > >> >DEFAULT group default qlen 1000 > >> >> link/ether 32:0f:0f:f0:60:f1 brd ff:ff:ff:ff:ff:ff > >> >> 26: r1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN > >mode > >> >DEFAULT group default qlen 1000 > >> >> link/ether 3e:5d:9a:4d:e7:7b brd ff:ff:ff:ff:ff:ff > >> >> > >> >> #devlink dev > >> >> pci/0002:01:00.0 > >> >> pci/0002:02:00.0 > >> >> pci/0002:03:00.0 > >> >> pci/0002:04:00.0 > >> >> pci/0002:05:00.0 > >> >> pci/0002:06:00.0 > >> >> pci/0002:07:00.0 > >> >> > >> >> ~# devlink port > >> >> pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller > >> >> 0 pfnum 1 vfnum 0 external false splittable false > >> >> pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller > >> >> 0 pfnum 1 vfnum 1 external false splittable false > >> >> pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller > >> >> 0 pfnum 1 vfnum 2 external false splittable false > >> >> pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller > >> >> 0 pfnum 1 vfnum 3 external false splittable false > >> >> > >> >> ----------- > >> >> v1-v2: > >> >> -Fixed build warnings. > >> >> -Address review comments provided by "Kalesh Anakkur Purayil". > >> >> > >> >> v2-v3: > >> >> - Used extack for error messages. > >> >> - As suggested reworked commit messages. > >> >> - Fixed sparse warning. > >> >> > >> >> v3-v4: > >> >> - Patch 2 & 3: Fixed coccinelle reported warnings. > >> >> - Patch 10: Added devlink port support. > >> >> > >> >> v4-v5: > >> >> - Patch 3: Removed devm_* usage in rvu_rep_create() > >> >> - Patch 3: Fixed build warnings. > >> >> > >> >> v5-v6: > >> >> - Addressed review comments provided by "Simon Horman". > >> >> - Added review tag. > >> >> > >> >> v6-v7: > >> >> - Rebased on top net-next branch. > >> >> > >> >> Geetha sowjanya (10): > >> >> octeontx2-pf: Refactoring RVU driver > >> >> octeontx2-pf: RVU representor driver > >> >> octeontx2-pf: Create representor netdev > >> >> octeontx2-pf: Add basic net_device_ops > >> >> octeontx2-af: Add packet path between representor and VF > >> >> octeontx2-pf: Get VF stats via representor > >> >> octeontx2-pf: Add support to sync link state between representor and > >> >> VFs > >> >> octeontx2-pf: Configure VF mtu via representor > >> >> octeontx2-pf: Add representors for sdp MAC > >> >> octeontx2-pf: Add devlink port support > >> >> > >> >> .../net/ethernet/marvell/octeontx2/Kconfig | 8 + > >> >> .../ethernet/marvell/octeontx2/af/Makefile | 3 +- > >> >> .../ethernet/marvell/octeontx2/af/common.h | 2 + > >> >> .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ > >> >> .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + > >> >> .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + > >> >> .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- > >> >> .../marvell/octeontx2/af/rvu_debugfs.c | 27 - > >> >> .../marvell/octeontx2/af/rvu_devlink.c | 6 + > >> >> .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 ++- > >> >> .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + > >> >> .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + > >> >> .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 ++++++++++++ > >> >> .../marvell/octeontx2/af/rvu_struct.h | 26 + > >> >> .../marvell/octeontx2/af/rvu_switch.c | 20 +- > >> >> .../ethernet/marvell/octeontx2/nic/Makefile | 2 + > >> >> .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- > >> >> .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- > >> >> .../marvell/octeontx2/nic/otx2_common.c | 56 +- > >> >> .../marvell/octeontx2/nic/otx2_common.h | 84 ++- > >> >> .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ > >> >> .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- > >> >> .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- > >> >> .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- > >> >> .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- > >> >> .../net/ethernet/marvell/octeontx2/nic/rep.c | 684 > >> >> ++++++++++++++++++ .../net/ethernet/marvell/octeontx2/nic/rep.h | > >> >> 53 ++ > >> >> 27 files changed, 1834 insertions(+), 227 deletions(-) create > >> >> mode > >> >> 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c > >> >> create mode 100644 > >> >> drivers/net/ethernet/marvell/octeontx2/nic/rep.c > >> >> create mode 100644 > >> >> drivers/net/ethernet/marvell/octeontx2/nic/rep.h > >> >> > >> >> -- > >> >> 2.25.1 > >> >>
>-----Original Message----- >From: Michal Swiatkowski <michal.swiatkowski@linux.intel.com> >Sent: Friday, July 5, 2024 2:04 PM >To: Geethasowjanya Akula <gakula@marvell.com> >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; kuba@kernel.org; >davem@davemloft.net; pabeni@redhat.com; edumazet@google.com; Sunil >Kovvuri Goutham <sgoutham@marvell.com>; Subbaraya Sundeep Bhatta ><sbhatta@marvell.com>; Hariprasad Kelam <hkelam@marvell.com> >Subject: Re: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU >representors > >On Thu, Jul 04, 2024 at 01: 48: 23PM +0000, Geethasowjanya Akula wrote: > > > >>-----Original Message----- > >From: Michal Swiatkowski ><michal. swiatkowski@ linux. intel. com> > >Sent: Thursday, July 4, 2024 12: 45 >PM >On Thu, Jul 04, 2024 at 01:48:23PM +0000, Geethasowjanya Akula wrote: >> >> >> >-----Original Message----- >> >From: Michal Swiatkowski <michal.swiatkowski@linux.intel.com> >> >Sent: Thursday, July 4, 2024 12:45 PM >> >To: Geethasowjanya Akula <gakula@marvell.com> >> >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; >> >kuba@kernel.org; davem@davemloft.net; pabeni@redhat.com; >> >edumazet@google.com; Sunil Kovvuri Goutham ><sgoutham@marvell.com>; >> >Subbaraya Sundeep Bhatta <sbhatta@marvell.com>; Hariprasad Kelam >> ><hkelam@marvell.com> >> >Subject: Re: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU >> >representors >> > >> >On Wed, Jul 03, 2024 at 02: 34: 03PM +0000, Geethasowjanya Akula >> >wrote: > > >> >> >-----Original Message----- > >From: Michal Swiatkowski >> ><michal. swiatkowski@ linux. intel. com> > >Sent: Wednesday, July 3, >> >2024 4: 14 PM On Wed, Jul 03, 2024 at 02:34:03PM +0000, >> >Geethasowjanya Akula wrote: >> >> >> >> >> >> >-----Original Message----- >> >> >From: Michal Swiatkowski <michal.swiatkowski@linux.intel.com> >> >> >Sent: Wednesday, July 3, 2024 4:14 PM >> >> >To: Geethasowjanya Akula <gakula@marvell.com> >> >> >Cc: netdev@vger.kernel.org; linux-kernel@vger.kernel.org; >> >> >kuba@kernel.org; davem@davemloft.net; pabeni@redhat.com; >> >> >edumazet@google.com; Sunil Kovvuri Goutham >> ><sgoutham@marvell.com>; >> >> >Subbaraya Sundeep Bhatta <sbhatta@marvell.com>; Hariprasad Kelam >> >> ><hkelam@marvell.com> >> >> >Subject: [EXTERNAL] Re: [net-next PATCH v7 00/10] Introduce RVU >> >> >representors On Fri, Jun 28, 2024 at 07:05:07PM +0530, Geetha >> >> >sowjanya >> >wrote: >> >> >> This series adds representor support for each rvu devices. >> >> >> When switchdev mode is enabled, representor netdev is registered >> >> >> for each rvu device. In implementation of representor model, one >> >> >> NIX HW LF with multiple SQ and RQ is reserved, where each RQ and >> >> >> SQ of the LF are mapped to a representor. A loopback channel is >> >> >> reserved to support packet path between representors and VFs. >> >> >> CN10K silicon supports 2 types of MACs, RPM and SDP. This patch >> >> >> set adds representor support for both RPM and SDP MAC interfaces. >> >> >> >> >> >> - Patch 1: Refactors and exports the shared service functions. >> >> >> - Patch 2: Implements basic representor driver. >> >> >> - Patch 3: Add devlink support to create representor netdevs that >> >> >> can be used to manage VFs. >> >> >> - Patch 4: Implements basec netdev_ndo_ops. >> >> >> - Patch 5: Installs tcam rules to route packets between representor and >> >> >> VFs. >> >> >> - Patch 6: Enables fetching VF stats via representor interface >> >> >> - Patch 7: Adds support to sync link state between representors and VFs >. >> >> >> - Patch 8: Enables configuring VF MTU via representor netdevs. >> >> >> - Patch 9: Add representors for sdp MAC. >> >> >> - Patch 10: Add devlink port support. >> >> >> >> >> >> Command to create VF representor #devlink dev eswitch set >> >> >> pci/0002:1c:00.0 mode switchdev VF representors are created for >> >> >> each VF when switch mode is set switchdev on representor PCI >> >> >> device >> >> > >> >> >Does it mean that VFs needs to be created before going to >> >> >switchdev mode? (in legacy mode). Keep in mind that in both >> >> >mellanox and ice driver assume that VFs are created after chaning >> >> >mode to switchdev (mode can't be changed if VFs). >> >> No. RVU representor driver implementation is similar to mellanox >> >> and ice >> >drivers. >> >> It assumes that VF gets created only after switchdev mode is enabled. >> >> Sorry, if above commit description is confusing. Will rewrite it. >> >> >> > >> >Ok, but why the rvu_rep_create() is called in switching mode to >> >switchdev function? In this function you are creating netdevs, only >> >for PF representor? It looks like it doesn't called from other >> >context in this patchset, so where the port representor netdevs for VFs are >created? >> > >> RVU representors for PF/VFs are created when switchdev mode is set, similar >to the bnxt and nfp drivers. >> rvu_rep_create() will create representors based on rep_cnt (which >> include both PFs and VFs count) >> > >Sorry, I don't understand now. You wrote in previous message that VFs should >be created after switchdev mode is enabled. Now you are writting that they are >created based on rep_cnt (so assuming VFs have been created before). > >What is the correct order? ># echo 1 x > sriov_numvfs ># devlink dev eswitch set pci/0000:ca:00.0 mode switchdev or # devlink dev >eswitch set pci/0000:ca:00.0 mode switchdev # echo 1 x > sriov_numvfs > >or you can create VFs before and after? From looking at the drivers code it >looks like driver bnxt supports that. Actually, we support VF creation before and after representor creation. Representor will not able send or receive any traffic untill corresponding representee is created. > >I am not familiar with nfp and bnxt driver but based on code in current kernel >version it looks like nfp creates representor when switching mode only for >physical and PF types, VF representor type is created when the VF is created. >Bnxt support VF representor creation before and after switching mode. In ice >port representors are created during VFs creation only. > >I am not against your solution, only pointing that it can lead to some problems. > Thanks for sharing the information. Geetha. >Thanks, >Michal > >> Thanks, >> Geetha. >> >> >Thanks, >> >Michal >> > >> >> > >> >> >Different order can be problematic. For example (AFAIK) kubernetes >> >> >scripts for switchdev assume that first is switching to switchdev >> >> >and VFs creation is done after that. >> >> > >> >> >Thanks, >> >> >Michal >> >> > >> >> >> # devlink dev eswitch set pci/0002:1c:00.0 mode switchdev # ip >> >> >> link show >> >> >> 25: r0p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >> >mode >> >> >DEFAULT group default qlen 1000 >> >> >> link/ether 32:0f:0f:f0:60:f1 brd ff:ff:ff:ff:ff:ff >> >> >> 26: r1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN >> >mode >> >> >DEFAULT group default qlen 1000 >> >> >> link/ether 3e:5d:9a:4d:e7:7b brd ff:ff:ff:ff:ff:ff >> >> >> >> >> >> #devlink dev >> >> >> pci/0002:01:00.0 >> >> >> pci/0002:02:00.0 >> >> >> pci/0002:03:00.0 >> >> >> pci/0002:04:00.0 >> >> >> pci/0002:05:00.0 >> >> >> pci/0002:06:00.0 >> >> >> pci/0002:07:00.0 >> >> >> >> >> >> ~# devlink port >> >> >> pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf >> >> >> controller >> >> >> 0 pfnum 1 vfnum 0 external false splittable false >> >> >> pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf >> >> >> controller >> >> >> 0 pfnum 1 vfnum 1 external false splittable false >> >> >> pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf >> >> >> controller >> >> >> 0 pfnum 1 vfnum 2 external false splittable false >> >> >> pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf >> >> >> controller >> >> >> 0 pfnum 1 vfnum 3 external false splittable false >> >> >> >> >> >> ----------- >> >> >> v1-v2: >> >> >> -Fixed build warnings. >> >> >> -Address review comments provided by "Kalesh Anakkur Purayil". >> >> >> >> >> >> v2-v3: >> >> >> - Used extack for error messages. >> >> >> - As suggested reworked commit messages. >> >> >> - Fixed sparse warning. >> >> >> >> >> >> v3-v4: >> >> >> - Patch 2 & 3: Fixed coccinelle reported warnings. >> >> >> - Patch 10: Added devlink port support. >> >> >> >> >> >> v4-v5: >> >> >> - Patch 3: Removed devm_* usage in rvu_rep_create() >> >> >> - Patch 3: Fixed build warnings. >> >> >> >> >> >> v5-v6: >> >> >> - Addressed review comments provided by "Simon Horman". >> >> >> - Added review tag. >> >> >> >> >> >> v6-v7: >> >> >> - Rebased on top net-next branch. >> >> >> >> >> >> Geetha sowjanya (10): >> >> >> octeontx2-pf: Refactoring RVU driver >> >> >> octeontx2-pf: RVU representor driver >> >> >> octeontx2-pf: Create representor netdev >> >> >> octeontx2-pf: Add basic net_device_ops >> >> >> octeontx2-af: Add packet path between representor and VF >> >> >> octeontx2-pf: Get VF stats via representor >> >> >> octeontx2-pf: Add support to sync link state between representor and >> >> >> VFs >> >> >> octeontx2-pf: Configure VF mtu via representor >> >> >> octeontx2-pf: Add representors for sdp MAC >> >> >> octeontx2-pf: Add devlink port support >> >> >> >> >> >> .../net/ethernet/marvell/octeontx2/Kconfig | 8 + >> >> >> .../ethernet/marvell/octeontx2/af/Makefile | 3 +- >> >> >> .../ethernet/marvell/octeontx2/af/common.h | 2 + >> >> >> .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ >> >> >> .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + >> >> >> .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + >> >> >> .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- >> >> >> .../marvell/octeontx2/af/rvu_debugfs.c | 27 - >> >> >> .../marvell/octeontx2/af/rvu_devlink.c | 6 + >> >> >> .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 ++- >> >> >> .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + >> >> >> .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + >> >> >> .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 ++++++++++++ >> >> >> .../marvell/octeontx2/af/rvu_struct.h | 26 + >> >> >> .../marvell/octeontx2/af/rvu_switch.c | 20 +- >> >> >> .../ethernet/marvell/octeontx2/nic/Makefile | 2 + >> >> >> .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- >> >> >> .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- >> >> >> .../marvell/octeontx2/nic/otx2_common.c | 56 +- >> >> >> .../marvell/octeontx2/nic/otx2_common.h | 84 ++- >> >> >> .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ >> >> >> .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- >> >> >> .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- >> >> >> .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- >> >> >> .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- >> >> >> .../net/ethernet/marvell/octeontx2/nic/rep.c | 684 >> >> >> ++++++++++++++++++ .../net/ethernet/marvell/octeontx2/nic/rep.h >> >> >> ++++++++++++++++++ | >> >> >> 53 ++ >> >> >> 27 files changed, 1834 insertions(+), 227 deletions(-) create >> >> >> mode >> >> >> 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c >> >> >> create mode 100644 >> >> >> drivers/net/ethernet/marvell/octeontx2/nic/rep.c >> >> >> create mode 100644 >> >> >> drivers/net/ethernet/marvell/octeontx2/nic/rep.h >> >> >> >> >> >> -- >> >> >> 2.25.1 >> >> >>
This series adds representor support for each rvu devices. When switchdev mode is enabled, representor netdev is registered for each rvu device. In implementation of representor model, one NIX HW LF with multiple SQ and RQ is reserved, where each RQ and SQ of the LF are mapped to a representor. A loopback channel is reserved to support packet path between representors and VFs. CN10K silicon supports 2 types of MACs, RPM and SDP. This patch set adds representor support for both RPM and SDP MAC interfaces. - Patch 1: Refactors and exports the shared service functions. - Patch 2: Implements basic representor driver. - Patch 3: Add devlink support to create representor netdevs that can be used to manage VFs. - Patch 4: Implements basec netdev_ndo_ops. - Patch 5: Installs tcam rules to route packets between representor and VFs. - Patch 6: Enables fetching VF stats via representor interface - Patch 7: Adds support to sync link state between representors and VFs . - Patch 8: Enables configuring VF MTU via representor netdevs. - Patch 9: Add representors for sdp MAC. - Patch 10: Add devlink port support. Command to create VF representor #devlink dev eswitch set pci/0002:1c:00.0 mode switchdev VF representors are created for each VF when switch mode is set switchdev on representor PCI device # devlink dev eswitch set pci/0002:1c:00.0 mode switchdev # ip link show 25: r0p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 32:0f:0f:f0:60:f1 brd ff:ff:ff:ff:ff:ff 26: r1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 3e:5d:9a:4d:e7:7b brd ff:ff:ff:ff:ff:ff #devlink dev pci/0002:01:00.0 pci/0002:02:00.0 pci/0002:03:00.0 pci/0002:04:00.0 pci/0002:05:00.0 pci/0002:06:00.0 pci/0002:07:00.0 ~# devlink port pci/0002:1c:00.0/0: type eth netdev r0p1v0 flavour pcipf controller 0 pfnum 1 vfnum 0 external false splittable false pci/0002:1c:00.0/1: type eth netdev r1p1v1 flavour pcivf controller 0 pfnum 1 vfnum 1 external false splittable false pci/0002:1c:00.0/2: type eth netdev r2p1v2 flavour pcivf controller 0 pfnum 1 vfnum 2 external false splittable false pci/0002:1c:00.0/3: type eth netdev r3p1v3 flavour pcivf controller 0 pfnum 1 vfnum 3 external false splittable false ----------- v1-v2: -Fixed build warnings. -Address review comments provided by "Kalesh Anakkur Purayil". v2-v3: - Used extack for error messages. - As suggested reworked commit messages. - Fixed sparse warning. v3-v4: - Patch 2 & 3: Fixed coccinelle reported warnings. - Patch 10: Added devlink port support. v4-v5: - Patch 3: Removed devm_* usage in rvu_rep_create() - Patch 3: Fixed build warnings. v5-v6: - Addressed review comments provided by "Simon Horman". - Added review tag. v6-v7: - Rebased on top net-next branch. Geetha sowjanya (10): octeontx2-pf: Refactoring RVU driver octeontx2-pf: RVU representor driver octeontx2-pf: Create representor netdev octeontx2-pf: Add basic net_device_ops octeontx2-af: Add packet path between representor and VF octeontx2-pf: Get VF stats via representor octeontx2-pf: Add support to sync link state between representor and VFs octeontx2-pf: Configure VF mtu via representor octeontx2-pf: Add representors for sdp MAC octeontx2-pf: Add devlink port support .../net/ethernet/marvell/octeontx2/Kconfig | 8 + .../ethernet/marvell/octeontx2/af/Makefile | 3 +- .../ethernet/marvell/octeontx2/af/common.h | 2 + .../net/ethernet/marvell/octeontx2/af/mbox.h | 74 ++ .../net/ethernet/marvell/octeontx2/af/npc.h | 1 + .../net/ethernet/marvell/octeontx2/af/rvu.c | 11 + .../net/ethernet/marvell/octeontx2/af/rvu.h | 30 +- .../marvell/octeontx2/af/rvu_debugfs.c | 27 - .../marvell/octeontx2/af/rvu_devlink.c | 6 + .../ethernet/marvell/octeontx2/af/rvu_nix.c | 81 ++- .../marvell/octeontx2/af/rvu_npc_fs.c | 5 + .../ethernet/marvell/octeontx2/af/rvu_reg.h | 4 + .../ethernet/marvell/octeontx2/af/rvu_rep.c | 464 ++++++++++++ .../marvell/octeontx2/af/rvu_struct.h | 26 + .../marvell/octeontx2/af/rvu_switch.c | 20 +- .../ethernet/marvell/octeontx2/nic/Makefile | 2 + .../ethernet/marvell/octeontx2/nic/cn10k.c | 4 +- .../ethernet/marvell/octeontx2/nic/cn10k.h | 2 +- .../marvell/octeontx2/nic/otx2_common.c | 56 +- .../marvell/octeontx2/nic/otx2_common.h | 84 ++- .../marvell/octeontx2/nic/otx2_devlink.c | 49 ++ .../ethernet/marvell/octeontx2/nic/otx2_pf.c | 305 +++++--- .../marvell/octeontx2/nic/otx2_txrx.c | 38 +- .../marvell/octeontx2/nic/otx2_txrx.h | 3 +- .../ethernet/marvell/octeontx2/nic/otx2_vf.c | 19 +- .../net/ethernet/marvell/octeontx2/nic/rep.c | 684 ++++++++++++++++++ .../net/ethernet/marvell/octeontx2/nic/rep.h | 53 ++ 27 files changed, 1834 insertions(+), 227 deletions(-) create mode 100644 drivers/net/ethernet/marvell/octeontx2/af/rvu_rep.c create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.c create mode 100644 drivers/net/ethernet/marvell/octeontx2/nic/rep.h