[net-next,00/10] devlink and mlx5: Introduce rate domains

Message

Tariq Toukan Feb. 13, 2025, 6:01 p.m. UTC

Hi,

This series introduces rate domains in devlink and mlx5
driver. Find detailed description by Cosmin below.

Regards,
Tariq


devlink objects support rate management for tx scheduling, which
involves maintaining a tree of rate nodes that corresponds to tx
schedulers in hardware. 'man devlink-rate' has the full details.

The tree of rate nodes is maintained per devlink object, protected by
the devlink lock.

There exists hardware capable of instantiating a tx scheduling tree
which spans multiple functions of the same physical device (and thus
devlink objects) and therefore the current API and locking scheme is
insufficient.

This patch series changes the devlink rate implementation and API to
allow supporting such hardware and managing tx scheduling trees across
multiple functions of a physical device.

Modeling this requires having devlink rate nodes with parents in other
devlink objects. A naive approach that relies on the current
one-lock-per-devlink model is impossible, as it would require in some
cases acquiring multiple devlink locks in the correct order.

The solution proposed is to move rates in a separate object named 'rate
domain'. Devlink objects create a private rate domain on init and
hardware that supports cross-function tx scheduling can switch to using
a shared rate domain for a set of devlink objects. Shared rate domains
have an additional lock serializing access to rate notes.
A new pair of devlink attributes is introduced for specifying a foreign
parent device as well as changes to the rate management devlink calls to
allow setting a rate node parent to the requested foreign parent device.
Finally, this API is used from mlx5 for NICs with the correct capability
bit to allow cross-function tx scheduling.

A note about net-shapers:
The net-shapers framework is completely orthogonal to this patch series.
net-shapers does shaping for tx queues, groups of queues and up to the
netdevice level. This patch series is for shaping across functions, so
it is strictly above the netdevice level in the shaping hierarchy.

This patch series was previously sent as an RFC ([1]).

Patches:

Small cleanup:
devlink: Remove unused param of devlink_rate_nodes_check

Introduce private rate domains:
devlink: Store devlink rates in a rate domain

Introduce rate domain locking (noop now as rate domains are private):
devlink: Serialize access to rate domains

Introduce shared rate domains and a global registry for them:
devlink: Introduce shared rate domains

Extend the devlink rate API with foreign parent devices:
devlink: Allow specifying parent device for rate commands
devlink: Allow rate node parents from other devlinks

Extends mlx5 implementation with the ability to share qos domains:
net/mlx5: qos: Introduce shared esw qos domains

Use the newly introduced stuff to support cross-function tx scheduling:
net/mlx5: qos: Support cross-esw tx scheduling
net/mlx5: qos: Init shared devlink rate domain

Finally, update documentation:
net/mlx5: Document devlink rates and cross-esw scheduling

[1] https://lore.kernel.org/netdev/[e-mail]/


Cosmin Ratiu (10):
  devlink: Remove unused param of devlink_rate_nodes_check
  devlink: Store devlink rates in a rate domain
  devlink: Serialize access to rate domains
  devlink: Introduce shared rate domains
  devlink: Allow specifying parent device for rate commands
  devlink: Allow rate node parents from other devlinks
  net/mlx5: qos: Introduce shared esw qos domains
  net/mlx5: qos: Support cross-esw tx scheduling
  net/mlx5: qos: Init shared devlink rate domain
  net/mlx5: Document devlink rates and cross-esw scheduling

 Documentation/netlink/specs/devlink.yaml      |  18 +-
 .../networking/devlink/devlink-port.rst       |   2 +
 Documentation/networking/devlink/mlx5.rst     |  33 +++
 .../net/ethernet/mellanox/mlx5/core/esw/qos.c | 144 ++++++++++--
 include/net/devlink.h                         |   8 +
 include/uapi/linux/devlink.h                  |   3 +
 net/devlink/core.c                            |  86 ++++++-
 net/devlink/dev.c                             |   6 +-
 net/devlink/devl_internal.h                   |  34 ++-
 net/devlink/netlink.c                         |  74 ++++--
 net/devlink/netlink_gen.c                     |  20 +-
 net/devlink/netlink_gen.h                     |   7 +
 net/devlink/rate.c                            | 217 +++++++++++++-----
 13 files changed, 548 insertions(+), 104 deletions(-)


base-commit: 8dbf0c7556454b52af91bae305ca71500c31495c