diff mbox series

[net-next,V9,14/14] net/mlx5: Add devlink subfunction port documentation

Message ID 20210121085237.137919-15-saeed@kernel.org (mailing list archive)
State Superseded
Delegated to: Netdev Maintainers
Headers show
Series [net-next,V9,01/14] devlink: Prepare code to fill multiple port function attributes | expand

Checks

Context Check Description
netdev/cover_letter warning Series does not have a cover letter
netdev/fixes_present success Link
netdev/patch_count success Link
netdev/tree_selection success Clearly marked for net-next
netdev/subject_prefix success Link
netdev/cc_maintainers warning 3 maintainers not CCed: leon@kernel.org linux-doc@vger.kernel.org corbet@lwn.net
netdev/source_inline success Was 0 now: 0
netdev/verify_signedoff success Link
netdev/module_param success Was 0 now: 0
netdev/build_32bit success Errors and warnings before: 0 this patch: 0
netdev/kdoc success Errors and warnings before: 0 this patch: 0
netdev/verify_fixes success Link
netdev/checkpatch success total: 0 errors, 0 warnings, 0 checks, 222 lines checked
netdev/build_allmodconfig_warn success Errors and warnings before: 0 this patch: 0
netdev/header_inline success Link
netdev/stable success Stable not CCed

Commit Message

Saeed Mahameed Jan. 21, 2021, 8:52 a.m. UTC
From: Parav Pandit <parav@nvidia.com>

Add documentation for subfunction management using devlink
port.

Signed-off-by: Parav Pandit <parav@nvidia.com>
Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
---
 .../device_drivers/ethernet/mellanox/mlx5.rst | 210 ++++++++++++++++++
 1 file changed, 210 insertions(+)

Comments

Samudrala, Sridhar Jan. 21, 2021, 8:59 p.m. UTC | #1
On 1/21/2021 12:52 AM, Saeed Mahameed wrote:
> From: Parav Pandit <parav@nvidia.com>
>
> Add documentation for subfunction management using devlink
> port.
>
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>
> ---
>   .../device_drivers/ethernet/mellanox/mlx5.rst | 210 ++++++++++++++++++
>   1 file changed, 210 insertions(+)
>
> diff --git a/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst b/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst
> index a5eb22793bb9..a1b32fcd0d76 100644
> --- a/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst
> +++ b/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst
> @@ -12,6 +12,8 @@ Contents
>   - `Enabling the driver and kconfig options`_
>   - `Devlink info`_
>   - `Devlink parameters`_
> +- `mlx5 subfunction`_
> +- `mlx5 port function`_
>   - `Devlink health reporters`_
>   - `mlx5 tracepoints`_
>   
> @@ -181,6 +183,214 @@ User command examples:
>         values:
>            cmode driverinit value true
>   
> +mlx5 subfunction
> +================
> +mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.
> +
> +A Subfunction has its own function capabilities and its own resources. This
> +means a subfunction has its own dedicated queues (txq, rxq, cq, eq). These
> +queues are neither shared nor stolen from the parent PCI function.
> +
> +When a subfunction is RDMA capable, it has its own QP1, GID table and rdma
> +resources neither shared nor stolen from the parent PCI function.
> +
> +A subfunction has a dedicated window in PCI BAR space that is not shared
> +with ther other subfunctions or the parent PCI function. This ensures that all
> +devices (netdev, rdma, vdpa etc.) of the subfunction accesses only assigned
> +PCI BAR space.
> +
> +A Subfunction supports eswitch representation through which it supports tc
> +offloads. The user configures eswitch to send/receive packets from/to
> +the subfunction port.
> +
> +Subfunctions share PCI level resources such as PCI MSI-X IRQs with
> +other subfunctions and/or with its parent PCI function.
> +
> +Example mlx5 software, system and device view::
> +
> +       _______
> +      | admin |
> +      | user  |----------
> +      |_______|         |
> +          |             |
> +      ____|____       __|______            _________________
> +     |         |     |         |          |                 |
> +     | devlink |     | tc tool |          |    user         |
> +     | tool    |     |_________|          | applications    |
> +     |_________|         |                |_________________|
> +           |             |                   |          |
> +           |             |                   |          |         Userspace
> + +---------|-------------|-------------------|----------|--------------------+
> +           |             |           +----------+   +----------+   Kernel
> +           |             |           |  netdev  |   | rdma dev |
> +           |             |           +----------+   +----------+
> +   (devlink port add/del |              ^               ^
> +    port function set)   |              |               |
> +           |             |              +---------------|
> +      _____|___          |              |        _______|_______
> +     |         |         |              |       | mlx5 class    |
> +     | devlink |   +------------+       |       |   drivers     |
> +     | kernel  |   | rep netdev |       |       |(mlx5_core,ib) |
> +     |_________|   +------------+       |       |_______________|
> +           |             |              |               ^
> +   (devlink ops)         |              |          (probe/remove)
> +  _________|________     |              |           ____|________
> + | subfunction      |    |     +---------------+   | subfunction |
> + | management driver|-----     | subfunction   |---|  driver     |
> + | (mlx5_core)      |          | auxiliary dev |   | (mlx5_core) |
> + |__________________|          +---------------+   |_____________|
> +           |                                            ^
> +  (sf add/del, vhca events)                             |
> +           |                                      (device add/del)
> +      _____|____                                    ____|________
> +     |          |                                  | subfunction |
> +     |  PCI NIC |---- activate/deactive events---->| host driver |
> +     |__________|                                  | (mlx5_core) |
> +                                                   |_____________|
> +
> +Subfunction is created using devlink port interface.
> +
> +- Change device to switchdev mode::
> +
> +    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
> +
> +- Add a devlink port of subfunction flaovur::
> +
> +    $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
> +    pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
> +      function:
> +        hw_addr 00:00:00:00:00:00 state inactive opstate detached
> +
> +- Show a devlink port of the subfunction::
> +
> +    $ devlink port show pci/0000:06:00.0/32768
> +    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
> +      function:
> +        hw_addr 00:00:00:00:00:00 state inactive opstate detached
> +
> +- Delete a devlink port of subfunction after use::
> +
> +    $ devlink port del pci/0000:06:00.0/32768
> +
> +mlx5 function attributes
> +========================
> +The mlx5 driver provides a mechanism to setup PCI VF/SF function attributes in
> +a unified way for SmartNIC and non-SmartNIC.
> +
> +This is supported only when the eswitch mode is set to switchdev. Port function
> +configuration of the PCI VF/SF is supported through devlink eswitch port.
> +
> +Port function attributes should be set before PCI VF/SF is enumerated by the
> +driver.
> +
> +MAC address setup
> +-----------------
> +mlx5 driver provides mechanism to setup the MAC address of the PCI VF/SF.
> +
> +The configured MAC address of the PCI VF/SF will be used by netdevice and rdma
> +device created for the PCI VF/SF.
> +
> +- Get the MAC address of the VF identified by its unique devlink port index::
> +
> +    $ devlink port show pci/0000:06:00.0/2
> +    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
> +      function:
> +        hw_addr 00:00:00:00:00:00
> +
> +- Set the MAC address of the VF identified by its unique devlink port index::
> +
> +    $ devlink port function set pci/0000:06:00.0/2 hw_addr 00:11:22:33:44:55
> +
> +    $ devlink port show pci/0000:06:00.0/2
> +    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
> +      function:
> +        hw_addr 00:11:22:33:44:55
> +
> +- Get the MAC address of the SF identified by its unique devlink port index::
> +
> +    $ devlink port show pci/0000:06:00.0/32768
> +    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
> +      function:
> +        hw_addr 00:00:00:00:00:00
> +
> +- Set the MAC address of the VF identified by its unique devlink port index::
> +
> +    $ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88
> +
> +    $ devlink port show pci/0000:06:00.0/32768
> +    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcivf pfnum 0 sfnum 88
> +      function:
> +        hw_addr 00:00:00:00:88:88
> +
> +SF state setup
> +--------------
> +To use the SF, the user must active the SF using the SF function state
> +attribute.
> +
> +- Get the state of the SF identified by its unique devlink port index::
> +
> +   $ devlink port show ens2f0npf0sf88
> +   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
> +     function:
> +       hw_addr 00:00:00:00:88:88 state inactive opstate detached
> +
> +- Activate the function and verify its state is active::
> +
> +   $ devlink port function set ens2f0npf0sf88 state active
> +
> +   $ devlink port show ens2f0npf0sf88
> +   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
> +     function:
> +       hw_addr 00:00:00:00:88:88 state active opstate detached
> +
> +Upon function activation, the PF driver instance gets the event from the device
> +that a particular SF was activated. It's the cue to put the device on bus, probe
> +it and instantiate the devlink instance and class specific auxiliary devices
> +for it.
> +
> +- Show the auxiliary device and port of the subfunction::
> +
> +    $ devlink dev show
> +    devlink dev show auxiliary/mlx5_core.sf.4
> +
> +    $ devlink port show auxiliary/mlx5_core.sf.4/1
> +    auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
> +
> +    $ rdma link show mlx5_0/1
> +    link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88
> +
> +    $ rdma dev show
> +    8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112
> +    13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
> +
> +- Subfunction auxiliary device and class device hierarchy::
> +
> +                 mlx5_core.sf.4
> +          (subfunction auxiliary device)
> +                       /\
> +                      /  \
> +                     /    \
> +                    /      \
> +                   /        \
> +      mlx5_core.eth.4     mlx5_core.rdma.4
> +     (sf eth aux dev)     (sf rdma aux dev)
> +         |                      |
> +         |                      |
> +      p0sf88                  mlx5_0
> +     (sf netdev)          (sf rdma device)

This picture seems to indicate that when SF is activated, a sub function 
auxiliary device is created
and when a driver is bound to that sub function aux device and probed, 2 
additional auxiliary devices are created.
Is this correct? Are all these auxiliary devices seen on the same aux bus?
Why do we need another sf eth aux device?


> +
> +Additionally, the SF port also gets the event when the driver attaches to the
> +auxiliary device of the subfunction. This results in changing the operational
> +state of the function. This provides visiblity to the user to decide when is it
> +safe to delete the SF port for graceful termination of the subfunction.
> +
> +- Show the SF port operational state::
> +
> +    $ devlink port show ens2f0npf0sf88
> +    pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
> +      function:
> +        hw_addr 00:00:00:00:88:88 state active opstate attached
> +
>   Devlink health reporters
>   ========================
>
Jason Gunthorpe Jan. 22, 2021, 12:11 a.m. UTC | #2
On Thu, Jan 21, 2021 at 12:59:55PM -0800, Samudrala, Sridhar wrote:

> > +                 mlx5_core.sf.4
> > +          (subfunction auxiliary device)
> > +                       /\
> > +                      /  \
> > +                     /    \
> > +                    /      \
> > +                   /        \
> > +      mlx5_core.eth.4     mlx5_core.rdma.4
> > +     (sf eth aux dev)     (sf rdma aux dev)
> > +         |                      |
> > +         |                      |
> > +      p0sf88                  mlx5_0
> > +     (sf netdev)          (sf rdma device)
> 
> This picture seems to indicate that when SF is activated, a sub
> function auxiliary device is created 

Yes

> and when a driver is bound to that sub function aux device and
> probed, 2 additional auxiliary devices are created.  

More than two, but yes

> Is this correct? Are all these auxiliary devices seen on the same
> aux bus?  

Yes

> Why do we need another sf eth aux device?

The first aux device represents the physical HW and mlx5_core binds to it,
the analog is like a pci_device.

The other aux devices represent the subsystem split of the mlx5 driver
- mlx5_core creates them and each subsystem in turn binds to the
mlx5_core driver. This already exists, and Intel will be doing this as
well whenever the RDMA driver is posted again..

Jason
Samudrala, Sridhar Jan. 23, 2021, 8:09 p.m. UTC | #3
On 1/21/2021 4:11 PM, Jason Gunthorpe wrote:
> On Thu, Jan 21, 2021 at 12:59:55PM -0800, Samudrala, Sridhar wrote:
>
>>> +                 mlx5_core.sf.4
>>> +          (subfunction auxiliary device)
>>> +                       /\
>>> +                      /  \
>>> +                     /    \
>>> +                    /      \
>>> +                   /        \
>>> +      mlx5_core.eth.4     mlx5_core.rdma.4
>>> +     (sf eth aux dev)     (sf rdma aux dev)
>>> +         |                      |
>>> +         |                      |
>>> +      p0sf88                  mlx5_0
>>> +     (sf netdev)          (sf rdma device)
>> This picture seems to indicate that when SF is activated, a sub
>> function auxiliary device is created
> Yes
>
>> and when a driver is bound to that sub function aux device and
>> probed, 2 additional auxiliary devices are created.
> More than two, but yes
>
>> Is this correct? Are all these auxiliary devices seen on the same
>> aux bus?
> Yes
>
>> Why do we need another sf eth aux device?
> The first aux device represents the physical HW and mlx5_core binds to it,
> the analog is like a pci_device.

OK. So looks like you are creating a function level aux device and a 
subsytem-level aux
device for each subsystem including ethernet.

>
> The other aux devices represent the subsystem split of the mlx5 driver
> - mlx5_core creates them and each subsystem in turn binds to the
> mlx5_core driver. This already exists, and Intel will be doing this as
> well whenever the RDMA driver is posted again..

Yes. I see that the intel RDMA patches are now submitted. We are 
creating an aux device to
expose RDMA functionality, but  not planning to create an aux device for 
ethernet subsystem
on a PF/SF as the function-level pci/aux device can represent the 
default ethernet.
Jason Gunthorpe Jan. 25, 2021, 1:06 p.m. UTC | #4
On Sat, Jan 23, 2021 at 12:09:15PM -0800, Samudrala, Sridhar wrote:

> > The other aux devices represent the subsystem split of the mlx5 driver
> > - mlx5_core creates them and each subsystem in turn binds to the
> > mlx5_core driver. This already exists, and Intel will be doing this as
> > well whenever the RDMA driver is posted again..
> 
> Yes. I see that the intel RDMA patches are now submitted. We are
> creating an aux device to expose RDMA functionality, but  not
> planning to create an aux device for ethernet subsystem on a PF/SF
> as the function-level pci/aux device can represent the default
> ethernet.

That is because the ethernet and shared code are all in the same
module

You may find this becomes inconvenient when you want to add something
like SF, where there is alot of merit to having uniformity in the
ethernet driver.

Jason
diff mbox series

Patch

diff --git a/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst b/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst
index a5eb22793bb9..a1b32fcd0d76 100644
--- a/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst
+++ b/Documentation/networking/device_drivers/ethernet/mellanox/mlx5.rst
@@ -12,6 +12,8 @@  Contents
 - `Enabling the driver and kconfig options`_
 - `Devlink info`_
 - `Devlink parameters`_
+- `mlx5 subfunction`_
+- `mlx5 port function`_
 - `Devlink health reporters`_
 - `mlx5 tracepoints`_
 
@@ -181,6 +183,214 @@  User command examples:
       values:
          cmode driverinit value true
 
+mlx5 subfunction
+================
+mlx5 supports subfunction management using devlink port (see :ref:`Documentation/networking/devlink/devlink-port.rst <devlink_port>`) interface.
+
+A Subfunction has its own function capabilities and its own resources. This
+means a subfunction has its own dedicated queues (txq, rxq, cq, eq). These
+queues are neither shared nor stolen from the parent PCI function.
+
+When a subfunction is RDMA capable, it has its own QP1, GID table and rdma
+resources neither shared nor stolen from the parent PCI function.
+
+A subfunction has a dedicated window in PCI BAR space that is not shared
+with ther other subfunctions or the parent PCI function. This ensures that all
+devices (netdev, rdma, vdpa etc.) of the subfunction accesses only assigned
+PCI BAR space.
+
+A Subfunction supports eswitch representation through which it supports tc
+offloads. The user configures eswitch to send/receive packets from/to
+the subfunction port.
+
+Subfunctions share PCI level resources such as PCI MSI-X IRQs with
+other subfunctions and/or with its parent PCI function.
+
+Example mlx5 software, system and device view::
+
+       _______
+      | admin |
+      | user  |----------
+      |_______|         |
+          |             |
+      ____|____       __|______            _________________
+     |         |     |         |          |                 |
+     | devlink |     | tc tool |          |    user         |
+     | tool    |     |_________|          | applications    |
+     |_________|         |                |_________________|
+           |             |                   |          |
+           |             |                   |          |         Userspace
+ +---------|-------------|-------------------|----------|--------------------+
+           |             |           +----------+   +----------+   Kernel
+           |             |           |  netdev  |   | rdma dev |
+           |             |           +----------+   +----------+
+   (devlink port add/del |              ^               ^
+    port function set)   |              |               |
+           |             |              +---------------|
+      _____|___          |              |        _______|_______
+     |         |         |              |       | mlx5 class    |
+     | devlink |   +------------+       |       |   drivers     |
+     | kernel  |   | rep netdev |       |       |(mlx5_core,ib) |
+     |_________|   +------------+       |       |_______________|
+           |             |              |               ^
+   (devlink ops)         |              |          (probe/remove)
+  _________|________     |              |           ____|________
+ | subfunction      |    |     +---------------+   | subfunction |
+ | management driver|-----     | subfunction   |---|  driver     |
+ | (mlx5_core)      |          | auxiliary dev |   | (mlx5_core) |
+ |__________________|          +---------------+   |_____________|
+           |                                            ^
+  (sf add/del, vhca events)                             |
+           |                                      (device add/del)
+      _____|____                                    ____|________
+     |          |                                  | subfunction |
+     |  PCI NIC |---- activate/deactive events---->| host driver |
+     |__________|                                  | (mlx5_core) |
+                                                   |_____________|
+
+Subfunction is created using devlink port interface.
+
+- Change device to switchdev mode::
+
+    $ devlink dev eswitch set pci/0000:06:00.0 mode switchdev
+
+- Add a devlink port of subfunction flaovur::
+
+    $ devlink port add pci/0000:06:00.0 flavour pcisf pfnum 0 sfnum 88
+    pci/0000:06:00.0/32768: type eth netdev eth6 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
+      function:
+        hw_addr 00:00:00:00:00:00 state inactive opstate detached
+
+- Show a devlink port of the subfunction::
+
+    $ devlink port show pci/0000:06:00.0/32768
+    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
+      function:
+        hw_addr 00:00:00:00:00:00 state inactive opstate detached
+
+- Delete a devlink port of subfunction after use::
+
+    $ devlink port del pci/0000:06:00.0/32768
+
+mlx5 function attributes
+========================
+The mlx5 driver provides a mechanism to setup PCI VF/SF function attributes in
+a unified way for SmartNIC and non-SmartNIC.
+
+This is supported only when the eswitch mode is set to switchdev. Port function
+configuration of the PCI VF/SF is supported through devlink eswitch port.
+
+Port function attributes should be set before PCI VF/SF is enumerated by the
+driver.
+
+MAC address setup
+-----------------
+mlx5 driver provides mechanism to setup the MAC address of the PCI VF/SF.
+
+The configured MAC address of the PCI VF/SF will be used by netdevice and rdma
+device created for the PCI VF/SF.
+
+- Get the MAC address of the VF identified by its unique devlink port index::
+
+    $ devlink port show pci/0000:06:00.0/2
+    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
+      function:
+        hw_addr 00:00:00:00:00:00
+
+- Set the MAC address of the VF identified by its unique devlink port index::
+
+    $ devlink port function set pci/0000:06:00.0/2 hw_addr 00:11:22:33:44:55
+
+    $ devlink port show pci/0000:06:00.0/2
+    pci/0000:06:00.0/2: type eth netdev enp6s0pf0vf1 flavour pcivf pfnum 0 vfnum 1
+      function:
+        hw_addr 00:11:22:33:44:55
+
+- Get the MAC address of the SF identified by its unique devlink port index::
+
+    $ devlink port show pci/0000:06:00.0/32768
+    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcisf pfnum 0 sfnum 88
+      function:
+        hw_addr 00:00:00:00:00:00
+
+- Set the MAC address of the VF identified by its unique devlink port index::
+
+    $ devlink port function set pci/0000:06:00.0/32768 hw_addr 00:00:00:00:88:88
+
+    $ devlink port show pci/0000:06:00.0/32768
+    pci/0000:06:00.0/32768: type eth netdev enp6s0pf0sf88 flavour pcivf pfnum 0 sfnum 88
+      function:
+        hw_addr 00:00:00:00:88:88
+
+SF state setup
+--------------
+To use the SF, the user must active the SF using the SF function state
+attribute.
+
+- Get the state of the SF identified by its unique devlink port index::
+
+   $ devlink port show ens2f0npf0sf88
+   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
+     function:
+       hw_addr 00:00:00:00:88:88 state inactive opstate detached
+
+- Activate the function and verify its state is active::
+
+   $ devlink port function set ens2f0npf0sf88 state active
+
+   $ devlink port show ens2f0npf0sf88
+   pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
+     function:
+       hw_addr 00:00:00:00:88:88 state active opstate detached
+
+Upon function activation, the PF driver instance gets the event from the device
+that a particular SF was activated. It's the cue to put the device on bus, probe
+it and instantiate the devlink instance and class specific auxiliary devices
+for it.
+
+- Show the auxiliary device and port of the subfunction::
+
+    $ devlink dev show
+    devlink dev show auxiliary/mlx5_core.sf.4
+
+    $ devlink port show auxiliary/mlx5_core.sf.4/1
+    auxiliary/mlx5_core.sf.4/1: type eth netdev p0sf88 flavour virtual port 0 splittable false
+
+    $ rdma link show mlx5_0/1
+    link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev p0sf88
+
+    $ rdma dev show
+    8: rocep6s0f1: node_type ca fw 16.29.0550 node_guid 248a:0703:00b3:d113 sys_image_guid 248a:0703:00b3:d112
+    13: mlx5_0: node_type ca fw 16.29.0550 node_guid 0000:00ff:fe00:8888 sys_image_guid 248a:0703:00b3:d112
+
+- Subfunction auxiliary device and class device hierarchy::
+
+                 mlx5_core.sf.4
+          (subfunction auxiliary device)
+                       /\
+                      /  \
+                     /    \
+                    /      \
+                   /        \
+      mlx5_core.eth.4     mlx5_core.rdma.4
+     (sf eth aux dev)     (sf rdma aux dev)
+         |                      |
+         |                      |
+      p0sf88                  mlx5_0
+     (sf netdev)          (sf rdma device)
+
+Additionally, the SF port also gets the event when the driver attaches to the
+auxiliary device of the subfunction. This results in changing the operational
+state of the function. This provides visiblity to the user to decide when is it
+safe to delete the SF port for graceful termination of the subfunction.
+
+- Show the SF port operational state::
+
+    $ devlink port show ens2f0npf0sf88
+    pci/0000:06:00.0/32768: type eth netdev ens2f0npf0sf88 flavour pcisf controller 0 pfnum 0 sfnum 88 external false splittable false
+      function:
+        hw_addr 00:00:00:00:88:88 state active opstate attached
+
 Devlink health reporters
 ========================