mbox series

[net-next,00/12] nfp: add support for multi-pf configuration

Message ID 20230724094821.14295-1-louis.peens@corigine.com (mailing list archive)
Headers show
Series nfp: add support for multi-pf configuration | expand

Message

Louis Peens July 24, 2023, 9:48 a.m. UTC 
This patch series is introducing multiple PFs for multiple ports NIC
assembled with NFP3800 chip. This is done since the NFP3800 can
support up to 4 PFs, and is more in-line with the modern expectation
that each port/netdev is associated with a unique PF.

For compatibility concern with NFP4000/6000 cards, and older management
firmware on NFP3800, multiple ports sharing single PF is still supported
with this change. Whether it's multi-PF setup or single-PF setup is
determined by management firmware, and driver will notify the
application firmware of the setup so that both are well handled.

* Patch 1/12 and 2/12 are to support new management firmware with bumped
  major version.
* Patch 3/12, 4/12, 5/12 adjust the application firmware loading and
  unloading mechanism since multi PFs share the same application
  firmware.
* Patch 6/12 is a small fix to avoid reclaiming resources by mistake in
  multi-PF setup.
* Patch 7/12 re-formats the symbols to communicate with application
  firmware to adapt multi-PF setup.
* Patch 8/12 applies one port/netdev per PF.
* Patch 9/12 is to support both single-PF and multi-PF setup by a
  configuration in application firmware.
* Patch 10/12, 11/12, 12/12 are some necessary adaption to use SR-IOV
  for multi-PF setup.

Tianyu Yuan (4):
  nsp: generate nsp command with variable nsp major version
  nfp: bump the nsp major version to support multi-PF
  nfp: apply one port per PF for multi-PF setup
  nfp: configure VF total count for each PF

Yinjun Zhang (8):
  nfp: change application firmware loading flow in multi-PF setup
  nfp: don't skip firmware loading when it's pxe firmware in running
  nfp: introduce keepalive mechanism for multi-PF setup
  nfp: avoid reclaiming resource mutex by mistake
  nfp: redefine PF id used to format symbols
  nfp: enable multi-PF in application firmware if supported
  nfp: configure VF split info into application firmware
  nfp: use absolute vf id for multi-PF case

 drivers/net/ethernet/netronome/nfp/abm/ctrl.c |   2 +-
 drivers/net/ethernet/netronome/nfp/abm/main.c |   2 +-
 drivers/net/ethernet/netronome/nfp/bpf/main.c |   2 +-
 .../net/ethernet/netronome/nfp/flower/main.c  |  19 +-
 drivers/net/ethernet/netronome/nfp/nfp_main.c | 227 ++++++++++++++++--
 drivers/net/ethernet/netronome/nfp/nfp_main.h |  28 +++
 .../net/ethernet/netronome/nfp/nfp_net_ctrl.h |   1 +
 .../net/ethernet/netronome/nfp/nfp_net_main.c | 166 ++++++++++---
 .../ethernet/netronome/nfp/nfp_net_sriov.c    |  39 ++-
 .../ethernet/netronome/nfp/nfp_net_sriov.h    |   5 +
 drivers/net/ethernet/netronome/nfp/nfp_port.c |   4 +-
 .../net/ethernet/netronome/nfp/nfpcore/nfp.h  |   4 +
 .../ethernet/netronome/nfp/nfpcore/nfp_dev.c  |   2 +
 .../ethernet/netronome/nfp/nfpcore/nfp_dev.h  |   1 +
 .../netronome/nfp/nfpcore/nfp_mutex.c         |  21 +-
 .../ethernet/netronome/nfp/nfpcore/nfp_nffw.h |   4 +
 .../ethernet/netronome/nfp/nfpcore/nfp_nsp.c  |  18 +-
 .../netronome/nfp/nfpcore/nfp_rtsym.c         |  16 +-
 drivers/net/ethernet/netronome/nfp/nic/main.c |   3 +-
 19 files changed, 474 insertions(+), 90 deletions(-)

Comments

Jakub Kicinski July 25, 2023, 12:01 a.m. UTC | #1 
On Mon, 24 Jul 2023 11:48:09 +0200 Louis Peens wrote:
> This patch series is introducing multiple PFs for multiple ports NIC
> assembled with NFP3800 chip. This is done since the NFP3800 can
> support up to 4 PFs, and is more in-line with the modern expectation
> that each port/netdev is associated with a unique PF.
> 
> For compatibility concern with NFP4000/6000 cards, and older management
> firmware on NFP3800, multiple ports sharing single PF is still supported
> with this change. Whether it's multi-PF setup or single-PF setup is
> determined by management firmware, and driver will notify the
> application firmware of the setup so that both are well handled.

So every PF will have its own devlink instance?
Can you show devlink dev info output?
Yinjun Zhang July 25, 2023, 1:28 a.m. UTC | #2 
On Tuesday, July 25, 2023 8:01 AM, Jakub Kicinski wrote:
> On Mon, 24 Jul 2023 11:48:09 +0200 Louis Peens wrote:
> > This patch series is introducing multiple PFs for multiple ports NIC
> > assembled with NFP3800 chip. This is done since the NFP3800 can
> > support up to 4 PFs, and is more in-line with the modern expectation
> > that each port/netdev is associated with a unique PF.
> >
> > For compatibility concern with NFP4000/6000 cards, and older management
> > firmware on NFP3800, multiple ports sharing single PF is still supported
> > with this change. Whether it's multi-PF setup or single-PF setup is
> > determined by management firmware, and driver will notify the
> > application firmware of the setup so that both are well handled.
> 
> So every PF will have its own devlink instance?
> Can you show devlink dev info output?

Yes, here it is:

pci/0000:01:00.0:
  driver nfp
  serial_number UKAAMDA2000-100122190023
  versions:
      fixed:
        board.id AMDA2000-1001
        board.rev 01
        board.manufacture UKA
        board.model schubert
      running:
        fw.mgmt 23.07-1
        fw.cpld 0x1000001
        fw.app sri-23.07.0-0
        chip.init AMDA-2000-1001  20230321150349
      stored:
        fw.mgmt 23.07-1
        fw.cpld 0x0
        chip.init AMDA-2000-1001  20230321150349
pci/0000:01:00.1:
  driver nfp
  serial_number UKAAMDA2000-100122190023
  versions:
      fixed:
        board.id AMDA2000-1001
        board.rev 01
        board.manufacture UKA
        board.model schubert
      running:
        fw.mgmt 23.07-1
        fw.cpld 0x1000001
        fw.app sri-23.07.0-0
        chip.init AMDA-2000-1001  20230321150349
      stored:
        fw.mgmt 23.07-1
        fw.cpld 0x0
        chip.init AMDA-2000-1001  20230321150349
Jakub Kicinski July 25, 2023, 6:59 p.m. UTC | #3 
On Tue, 25 Jul 2023 01:28:34 +0000 Yinjun Zhang wrote:
> On Tuesday, July 25, 2023 8:01 AM, Jakub Kicinski wrote:
> > On Mon, 24 Jul 2023 11:48:09 +0200 Louis Peens wrote:  
> > > This patch series is introducing multiple PFs for multiple ports NIC
> > > assembled with NFP3800 chip. This is done since the NFP3800 can
> > > support up to 4 PFs, and is more in-line with the modern expectation
> > > that each port/netdev is associated with a unique PF.
> > >
> > > For compatibility concern with NFP4000/6000 cards, and older management
> > > firmware on NFP3800, multiple ports sharing single PF is still supported
> > > with this change. Whether it's multi-PF setup or single-PF setup is
> > > determined by management firmware, and driver will notify the
> > > application firmware of the setup so that both are well handled.  
> > 
> > So every PF will have its own devlink instance?
> > Can you show devlink dev info output?  
> 
> Yes, here it is:

>   serial_number UKAAMDA2000-100122190023

>   serial_number UKAAMDA2000-100122190023

Since it's clearly a single ASIC shouldn't it have a single devlink
instance?
Yinjun Zhang July 26, 2023, 2 a.m. UTC | #4 
On Wednesday, July 26, 2023 3:00 AM, Jakub Kicinski wrote:
> On Tue, 25 Jul 2023 01:28:34 +0000 Yinjun Zhang wrote:
> > On Tuesday, July 25, 2023 8:01 AM, Jakub Kicinski wrote:
> > > On Mon, 24 Jul 2023 11:48:09 +0200 Louis Peens wrote:
> > > > This patch series is introducing multiple PFs for multiple ports NIC
> > > > assembled with NFP3800 chip. This is done since the NFP3800 can
> > > > support up to 4 PFs, and is more in-line with the modern expectation
> > > > that each port/netdev is associated with a unique PF.
> > > >
> > > > For compatibility concern with NFP4000/6000 cards, and older
> management
> > > > firmware on NFP3800, multiple ports sharing single PF is still supported
> > > > with this change. Whether it's multi-PF setup or single-PF setup is
> > > > determined by management firmware, and driver will notify the
> > > > application firmware of the setup so that both are well handled.
> > >
> > > So every PF will have its own devlink instance?
> > > Can you show devlink dev info output?
> >
> > Yes, here it is:
> 
> >   serial_number UKAAMDA2000-100122190023
> 
> >   serial_number UKAAMDA2000-100122190023
> 
> Since it's clearly a single ASIC shouldn't it have a single devlink
> instance?

But there're more than one PCI device now. Isn't it universal implementation
to register a devlink for each PCI device?
Jakub Kicinski July 26, 2023, 4:17 a.m. UTC | #5 
On Wed, 26 Jul 2023 02:00:30 +0000 Yinjun Zhang wrote:
> > >   serial_number UKAAMDA2000-100122190023  
> >   
> > >   serial_number UKAAMDA2000-100122190023  
> > 
> > Since it's clearly a single ASIC shouldn't it have a single devlink
> > instance?  
> 
> But there're more than one PCI device now. Isn't it universal implementation
> to register a devlink for each PCI device?

It's only the prevailing implementation because people are too lazy to
implement things correctly, if you ask me. devlink doesn't have the
ability to bind to multiple bus devices, that would need to be
addressed.
Yinjun Zhang July 26, 2023, 7:28 a.m. UTC | #6 
On Wednesday, July 26, 2023 12:17 PM, Jakub Kicinski wrote:
> On Wed, 26 Jul 2023 02:00:30 +0000 Yinjun Zhang wrote:
> > > >   serial_number UKAAMDA2000-100122190023
> > >
> > > >   serial_number UKAAMDA2000-100122190023
> > >
> > > Since it's clearly a single ASIC shouldn't it have a single devlink
> > > instance?
> >
> > But there're more than one PCI device now. Isn't it universal
> implementation
> > to register a devlink for each PCI device?
> 
> It's only the prevailing implementation because people are too lazy to
> implement things correctly, if you ask me. devlink doesn't have the
> ability to bind to multiple bus devices, that would need to be
> addressed.

It sounds like a separate topic to refine devlink infrastructure.