diff mbox series

[RFC,net-next,01/28] docs: networking: ti: add driver doc for CPSW Proxy Client

Message ID 20240518124234.2671651-2-s-vadapalli@ti.com (mailing list archive)
State New
Headers show
Series Add CPSW Proxy Client driver | expand

Commit Message

Siddharth Vadapalli May 18, 2024, 12:42 p.m. UTC
The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on
a remote core to enable Ethernet functionality for applications running
on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW
Ethernet Switch on the SoC and acts as the Server, offering services to
Clients running on various cores.

Signed-off-by: Siddharth Vadapalli <s-vadapalli@ti.com>
---
 .../ethernet/ti/cpsw_proxy_client.rst         | 182 ++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100644 Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst

Comments

Andrew Lunn May 19, 2024, 3:31 p.m. UTC | #1
On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote:
> The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on
> a remote core to enable Ethernet functionality for applications running
> on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW
> Ethernet Switch on the SoC and acts as the Server, offering services to
> Clients running on various cores.

I'm not sure we as a community what this architecture. We want Linux
to be driving the hardware, not firmware. So expect linux to be
running the server.

> +The "am65-cpsw-nuss.c" driver in Linux at:
> +drivers/net/ethernet/ti/am65-cpsw-nuss.c
> +provides Ethernet functionality for applications on Linux.
> +It also handles both the control-path and data-path, namely:
> +Control => Configuration of the CPSW Peripheral
> +Data => Configuration of the DMA Channels to transmit/receive data

So nuss is capable of controlling the hardware. nuss has an upper
interface which is switchdev, and a lower interface which somehow acts
on the hardware, maybe invoking RPCs into the firmware?

So it is not too big a step to put the server functionality in Linux,
on top of the nuss driver.

Now take a step backwards. This concept of a switch with different
CPUs attached to it is nothing special.

Some Marvell switches have a Z80 connected to a dedicated port of the
switch. You can run applications on that Z80. Those applications might
be as simple as spanning tree, so you can have a white box 8 port
ethernet switch without needing an external CPU. But there is an SDK,
you could run any sort of application on the Z80.

The microchip LAN996x switch has a Cortex A7 CPU, but also a PCIe
interface. Linux can control the switch via the PCIe interface, but
there could also be applications running on the Cortex.

Look at the Broadcom BCM89586M:
https://docs.broadcom.com/doc/89586M-PB

It is similar to the microchip device, an M7 CPU, and a PCIe
interface. It seems like a Linux host could be controlling the switch
via PCIe, and applications running on the M7.

I expect there are more examples, but you get the idea.

A completely different angle to look at is VF/PF and eswitches. A
server style CPU running virtual machines, a VM getting a VF passed
through to it. This is not something i know much about, so we might
need to pull in some data centre specialists. But we have a different
CPU, a virtual CPU, making use of part of the switch. Its the same
concept really.

My main point is, please start with an abstract view of the problem. A
lot of the solution should be generic, it can be applied to all these
devices. And then there probably needs to be a small part which is
specific to TI devices. It could be parts of the solutions already
exist, e.g. VF/PF have port represents, which might be a useful
concept here as well. Switchdev exists and provides a generic
interface for configuring switches...

	Andrew
Siddharth Vadapalli May 21, 2024, 12:12 p.m. UTC | #2
On Sun, May 19, 2024 at 05:31:16PM +0200, Andrew Lunn wrote:
> On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote:
> > The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on
> > a remote core to enable Ethernet functionality for applications running
> > on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW
> > Ethernet Switch on the SoC and acts as the Server, offering services to
> > Clients running on various cores.
> 
> I'm not sure we as a community what this architecture. We want Linux
> to be driving the hardware, not firmware. So expect linux to be
> running the server.
> 
> > +The "am65-cpsw-nuss.c" driver in Linux at:
> > +drivers/net/ethernet/ti/am65-cpsw-nuss.c
> > +provides Ethernet functionality for applications on Linux.
> > +It also handles both the control-path and data-path, namely:
> > +Control => Configuration of the CPSW Peripheral
> > +Data => Configuration of the DMA Channels to transmit/receive data
> 
> So nuss is capable of controlling the hardware. nuss has an upper
> interface which is switchdev, and a lower interface which somehow acts
> on the hardware, maybe invoking RPCs into the firmware?
> 
> So it is not too big a step to put the server functionality in Linux,
> on top of the nuss driver.

Andrew,

Thank you for reviewing the patch and sharing your feedback. While I
have come across other Switch Designs / Architecture, I am yet to go
through the one you have mentioned below. I will go through it in detail
and will follow up with my understanding in a future reply. This reply
is intended to be an acknowledgment that I have read your feedback.
I also wanted to clarify the use-case which this series targets. The
requirements of the use-case are:
1. Independent Ethernet Switch functionality: Switch operation and
configuration when Linux is not functional (Fast startup, Low Power
Mode, Safety use-cases).
2. Dynamic Ethernet Switch configuration changes performed based on the
applications which run on various cores.

[...]

Regards,
Siddharth.
Andrew Lunn May 21, 2024, 1:07 p.m. UTC | #3
> Andrew,
> 
> Thank you for reviewing the patch and sharing your feedback. While I
> have come across other Switch Designs / Architecture, I am yet to go
> through the one you have mentioned below. I will go through it in detail
> and will follow up with my understanding in a future reply. This reply
> is intended to be an acknowledgment that I have read your feedback.
> I also wanted to clarify the use-case which this series targets. The
> requirements of the use-case are:
> 1. Independent Ethernet Switch functionality: Switch operation and
> configuration when Linux is not functional (Fast startup, Low Power
> Mode, Safety use-cases).
> 2. Dynamic Ethernet Switch configuration changes performed based on the
> applications which run on various cores.

Please make sure these requirements are clearly stated in the design.

The support for switches in Linux has initially come from big data
centre switches, and smaller SOHO switches you found in OpenWRT class
devices. The switchdev model has worked well so far for these use
cases. However, i do understand you have additional requirements.

Ideally we want to extend the existing model to support additional use
cases, not create a second parallel model. And we want a vendor
agnostic extensions of the switchdev model, something which all
automotive vendors can use, and non-automotive systems which have a
similar architecture.

	Andrew
diff mbox series

Patch

diff --git a/Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst b/Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst
new file mode 100644
index 000000000000..46bff67b3446
--- /dev/null
+++ b/Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst
@@ -0,0 +1,182 @@ 
+.. SPDX-License-Identifier: GPL-2.0-only or MIT
+
+==========================================
+Texas Instruments CPSW Proxy Client driver
+==========================================
+
+Introduction
+============
+
+The CPSW (Common Platform Switch) Ethernet Switch on TI's K3 SoCs provides
+Ethernet functionality. There may be multiple instances of CPSW on a single
+SoC. The term "CPSWnG" is used to indicate the number of MAC Ports supported
+by a specific instance of CPSW. CPSWnG indicates that the peripheral has
+(n-1) MAC Ports and 1 Host Port. Examples of existing instances are:
+CPSW2G => 1 MAC Port and 1 Host Port
+CPSW3G => 2 MAC Ports and 1 Host Port
+CPSW5G => 4 MAC Ports and 1 Host Port
+CPSW9G => 8 MAC Ports and 1 Host Port
+
+The presence of 2 or more MAC Ports implies that Hardware Switching can
+be enabled between the MAC Ports if required.
+
+The "am65-cpsw-nuss.c" driver in Linux at:
+drivers/net/ethernet/ti/am65-cpsw-nuss.c
+provides Ethernet functionality for applications on Linux.
+It also handles both the control-path and data-path, namely:
+Control => Configuration of the CPSW Peripheral
+Data => Configuration of the DMA Channels to transmit/receive data
+
+The aforementioned configuration supports use-cases where all applications
+which require Ethernet functionality are only running on Linux.
+
+However, there are use-cases where applications running on different
+Operating Systems across multiple cores on the SoC require Ethernet
+functionality. Such use-cases can be supported by implementing a
+Client-Server model to share the data-path among Clients while the Server
+owns the control-path.
+
+On TI's K3 SoCs (J721E, J7200 and J784S4 in particular), the Ethernet Switch
+Firmware (EthFw) running on the MAIN R5F core acts as the Server and
+configures the CPSWnG instance (CPSW5G on J7200 and CPSW9G on J721E, J784S4)
+of the CPSW Ethernet Switch on the SoC. The Clients running on various cores
+communicate with EthFw via RPMsg (Remote Processor Messaging) to request
+resource allocation details during initialization, followed by requesting
+EthFw to enable features supported by CPSW based on the features required
+by the applications running on the respective cores.
+
+EthFw handles requests from the Clients and evaluates them before configuring
+CPSW based on the request. Since no Client is actually in control of CPSW and
+only requests EthFw for configuring CPSW, EthFw acts as the proxy for the
+Clients. Thus, the Linux Client which interfaces with EthFw is named:
+CPSW Proxy Client
+
+The data-path for the CPSW Proxy Client driver remains identical to the
+"am65-cpsw-nuss.c" driver which happens to be DMA. It is only the control-path
+that is different.
+
+Client-Server discovery occurs over the RPMsg-Bus. EthFw announces its
+RPMsg Endpoint name over the RPMsg-Bus. The CPSW Proxy Client driver
+registers itself with the Linux RPMsg framework to be probed for the same
+Endpoint name. Following probe, the Linux Client driver begins communicating
+with EthFw and queries details of the resources available for the Linux Client.
+
+Terminology
+===========
+
+Virtual Port
+        A Virtual Port refers to the Software View of an Ethernet MAC Port.
+        There are two types of Virtual Ports:
+        1. Virtual MAC Only Port
+        2. Virtual Switch Port
+
+Virtual MAC Only Port
+        A Virtual MAC only Port refers to a dedicated physical MAC Port for
+        a Client. This corresponds to MAC Mode of operation in Ethernet
+        Terminology. All traffic sent to or received from the Physical
+        MAC Port is that of the Client to which the Virtual MAC Only Port
+        has been allocated.
+
+Virtual Switch Port
+        A Virtual Switch Port refers to a group of physical MAC ports with
+        Switching enabled across them. This implies that any traffic sent
+        to the Port from a Client could potentially exit a Physical MAC
+        Port along with the traffic from other Clients. Similarly, the traffic
+        received on the Port by a Client could have potentially ingressed
+        on a Physical MAC Port along with the traffic meant for other Clients.
+        While the ALE (Address Lookup Engine) handles segregating the traffic,
+        and the CPSW Ethernet Switch places traffic on dedicated RX DMA Flows
+        meant for a single Client, it is worth noting that the bandwidths
+        of the Physical MAC Port are shared by Clients when traffic is sent to
+        or received from a Virtual Switch Port.
+
+Network Interface
+        The user-visible interface in Linux userspace exposed to applications
+        that serves as the entry/exit point for traffic to/from the Virtual
+        Ports. A single network interface (ethX) maps to either a Virtual
+        MAC Only Port or a Virtual Switch Port.
+
+C2S
+        RPMsg source is Client and destination is Server.
+
+S2C
+        RPMsg source is Server and destination is Client.
+
+Initialization Sequence
+=======================
+
+The sequence of message exchanges between the Client driver and EthFw starting
+from the driver probe and ending with the interfaces being brought up is as
+follows:
+1. C2S ETHFW_VIRT_PORT_INFO requesting details of Virtual Ports available
+   for the Linux Client.
+2. S2C response containing requested details
+3. C2S ETHFW_VIRT_PORT_ATTACH request for each Virtual Port allocated during
+   step 2.
+4. S2C response containing details of the MTU Size, number of Tx DMA Channels
+   and RX DMA Flows for the specified Virtual Port. The *Features* associated
+   with the Virtual Port are also shared such as Multicast Filtering capability.
+5. C2S ETHFW_ALLOC_RX request for each RX DMA Flow for a Virtual Port.
+6. S2C response containing details of the RX PSI-L Thread ID, Flow base and
+   Flow offset.
+7. C2S ETHFW_ALLOC_TX request for each TX DMA Channel for a Virtual Port.
+8. S2C response containing details of the TX PSI-L Thread ID.
+9. C2S ETHFW_ALLOC_MAC request for each Virtual Port.
+10. S2C response containing the MAC Address corresponding to the Virtual Port.
+11. C2S ETHFW_MAC_REGISTER request for each Virtual Port with the MAC Address
+    allocated in step 10. This is necessary to steer packets that ingress on
+    the MAC Ports of CPSW onto the RX DMA Flow for the Virtual Port in order
+    to allow the Client to receive the packets.
+12. S2C response indicating status of request.
+13. C2S ETHFW_IPv4_REGISTER request *only* for Virtual Switch Port interface.
+    The IPv4 address assigned to the "ethX" network interface in Linux
+    corresponding to the Virtual Switch Port interface has to be registered
+    with EthFw. This is due to the reason that all Broadcast requests including
+    ARP requests received by the MAC Ports corresponding to the Virtual Switch
+    Port are consumed solely be EthFw. Such traffic is sent to Clients by
+    alternate methods. Therefore EthFw needs to know the IPv4 address for the
+    "ethX" network interface in Linux in order to automatically respond to
+    ARP requests, thereby enabling Unicast communication.
+14. S2C response indicating status of request.
+15. C2S ETHFW_MCAST_FILTER_ADD request to register the Multicast Addresses
+    associated with the network interface corresponding to the Virtual Port
+    which has the Multicast Filtering capability.
+16. S2C response indicating status of request.
+17. C2S ETHFW_MCAST_FILTER_DEL request to deregister the Multicast Addresses
+    associated with the network interface corresponding to the Virtual Port
+    which has the Multicast Filtering capability.
+18. S2C response indicating status of request.
+
+Shutdown Sequence
+=================
+
+The sequence of message exchanges between the Client driver and EthFw on module
+removal are as follows:
+1. C2S ETHFW_MAC_DEREGISTER request to deregister the MAC Address for each
+   Virtual Port.
+2. S2C response indicating status of request.
+3. C2S ETHFW_MCAST_FILTER_DEL request to deregister the Multicast Addresses
+   associated with the network interface corresponding to the Virtual Port
+   which has the Multicast Filtering capability.
+4. S2C response indicating status of request.
+5. C2S ETHFW_FREE_MAC request to release the MAC Address allocated to each
+   Virtual Port.
+6. S2C response indicating status of request.
+7. C2S ETHFW_FREE_TX request to release the TX DMA Channel for each TX Channel
+   for every Virtual Port.
+8. S2C response indicating status of request.
+9. C2S ETHFW_FREE_RX request to release the RX DMA Flow for each RX Channel
+   for every Virtual Port.
+10. S2C response indicating status of request.
+11. C2S ETHFW_VIRT_PORT_DETACH request to release each Virtual Port.
+12. S2C response indicating status of request.
+
+Features Supported
+==================
+
+The set of features supported in addition to providing basic Ethernet
+Functionality are:
+1. Multicast Filtering
+2. Determining Link Status of the network interface corresponding to the
+   Virtual MAC Only port via ethtool.
+3. Interrupt Pacing/Coalescing