| Message ID | 20240518124234.2671651-2-s-vadapalli@ti.com (mailing list archive) |
|---|---|
| State | RFC |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | Add CPSW Proxy Client driver | expand |
On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote: > The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on > a remote core to enable Ethernet functionality for applications running > on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW > Ethernet Switch on the SoC and acts as the Server, offering services to > Clients running on various cores. I'm not sure we as a community what this architecture. We want Linux to be driving the hardware, not firmware. So expect linux to be running the server. > +The "am65-cpsw-nuss.c" driver in Linux at: > +drivers/net/ethernet/ti/am65-cpsw-nuss.c > +provides Ethernet functionality for applications on Linux. > +It also handles both the control-path and data-path, namely: > +Control => Configuration of the CPSW Peripheral > +Data => Configuration of the DMA Channels to transmit/receive data So nuss is capable of controlling the hardware. nuss has an upper interface which is switchdev, and a lower interface which somehow acts on the hardware, maybe invoking RPCs into the firmware? So it is not too big a step to put the server functionality in Linux, on top of the nuss driver. Now take a step backwards. This concept of a switch with different CPUs attached to it is nothing special. Some Marvell switches have a Z80 connected to a dedicated port of the switch. You can run applications on that Z80. Those applications might be as simple as spanning tree, so you can have a white box 8 port ethernet switch without needing an external CPU. But there is an SDK, you could run any sort of application on the Z80. The microchip LAN996x switch has a Cortex A7 CPU, but also a PCIe interface. Linux can control the switch via the PCIe interface, but there could also be applications running on the Cortex. Look at the Broadcom BCM89586M: https://docs.broadcom.com/doc/89586M-PB It is similar to the microchip device, an M7 CPU, and a PCIe interface. It seems like a Linux host could be controlling the switch via PCIe, and applications running on the M7. I expect there are more examples, but you get the idea. A completely different angle to look at is VF/PF and eswitches. A server style CPU running virtual machines, a VM getting a VF passed through to it. This is not something i know much about, so we might need to pull in some data centre specialists. But we have a different CPU, a virtual CPU, making use of part of the switch. Its the same concept really. My main point is, please start with an abstract view of the problem. A lot of the solution should be generic, it can be applied to all these devices. And then there probably needs to be a small part which is specific to TI devices. It could be parts of the solutions already exist, e.g. VF/PF have port represents, which might be a useful concept here as well. Switchdev exists and provides a generic interface for configuring switches... Andrew
On Sun, May 19, 2024 at 05:31:16PM +0200, Andrew Lunn wrote: > On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote: > > The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on > > a remote core to enable Ethernet functionality for applications running > > on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW > > Ethernet Switch on the SoC and acts as the Server, offering services to > > Clients running on various cores. > > I'm not sure we as a community what this architecture. We want Linux > to be driving the hardware, not firmware. So expect linux to be > running the server. > > > +The "am65-cpsw-nuss.c" driver in Linux at: > > +drivers/net/ethernet/ti/am65-cpsw-nuss.c > > +provides Ethernet functionality for applications on Linux. > > +It also handles both the control-path and data-path, namely: > > +Control => Configuration of the CPSW Peripheral > > +Data => Configuration of the DMA Channels to transmit/receive data > > So nuss is capable of controlling the hardware. nuss has an upper > interface which is switchdev, and a lower interface which somehow acts > on the hardware, maybe invoking RPCs into the firmware? > > So it is not too big a step to put the server functionality in Linux, > on top of the nuss driver. Andrew, Thank you for reviewing the patch and sharing your feedback. While I have come across other Switch Designs / Architecture, I am yet to go through the one you have mentioned below. I will go through it in detail and will follow up with my understanding in a future reply. This reply is intended to be an acknowledgment that I have read your feedback. I also wanted to clarify the use-case which this series targets. The requirements of the use-case are: 1. Independent Ethernet Switch functionality: Switch operation and configuration when Linux is not functional (Fast startup, Low Power Mode, Safety use-cases). 2. Dynamic Ethernet Switch configuration changes performed based on the applications which run on various cores. [...] Regards, Siddharth.
> Andrew, > > Thank you for reviewing the patch and sharing your feedback. While I > have come across other Switch Designs / Architecture, I am yet to go > through the one you have mentioned below. I will go through it in detail > and will follow up with my understanding in a future reply. This reply > is intended to be an acknowledgment that I have read your feedback. > I also wanted to clarify the use-case which this series targets. The > requirements of the use-case are: > 1. Independent Ethernet Switch functionality: Switch operation and > configuration when Linux is not functional (Fast startup, Low Power > Mode, Safety use-cases). > 2. Dynamic Ethernet Switch configuration changes performed based on the > applications which run on various cores. Please make sure these requirements are clearly stated in the design. The support for switches in Linux has initially come from big data centre switches, and smaller SOHO switches you found in OpenWRT class devices. The switchdev model has worked well so far for these use cases. However, i do understand you have additional requirements. Ideally we want to extend the existing model to support additional use cases, not create a second parallel model. And we want a vendor agnostic extensions of the switchdev model, something which all automotive vendors can use, and non-automotive systems which have a similar architecture. Andrew
On Sun, May 19, 2024 at 05:31:16PM +0200, Andrew Lunn wrote: Andrew, I have spent time going through your feedback, trying to understand your suggestions. This email is the complete reply corresponding to my earlier reply at: https://lore.kernel.org/r/0b0c1b07-756e-439e-bfc5-53824fd2a61c@ti.com/ which was simply meant to serve as an acknowledgement that I have seen your email. > On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote: > > The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on > > a remote core to enable Ethernet functionality for applications running > > on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW > > Ethernet Switch on the SoC and acts as the Server, offering services to > > Clients running on various cores. > > I'm not sure we as a community what this architecture. We want Linux > to be driving the hardware, not firmware. So expect linux to be > running the server. Due to the use-case requirements, Linux cannot be the server. Some of the requirements are: 1. Fast startup and configuration of CPSW independent of Linux and Other OS running on any of the cores on the SoC. The configuration of CPSW has to be performed in parallel while the Bootloader starts Linux. 2. CPSW has to be functional and configurable even when Linux has been suspended. One of the non-Linux Clients happens to be the AUTOSAR Client which continues exchanging network data via CPSW even when Linux has been suspended. So the server has to be functional even then, in order to cater to the AUTOSAR Client's requests to configure CPSW. CPSW's configuration is not static in the sense that it gets programmed and will no longer be modified. Therefore the server has to be functional at all times to update CPSW's configuration based on the demands of any of the Clients. For more details about the Ethernet Switch Firmware (EthFw) and the set of Clients running on remote cores, please refer: https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/09_02_00_05/exports/docs/ethfw/docs/user_guide/ethfw_c_ug_top.html#ethfw_remote_clients > > > +The "am65-cpsw-nuss.c" driver in Linux at: > > +drivers/net/ethernet/ti/am65-cpsw-nuss.c > > +provides Ethernet functionality for applications on Linux. > > +It also handles both the control-path and data-path, namely: > > +Control => Configuration of the CPSW Peripheral > > +Data => Configuration of the DMA Channels to transmit/receive data > > So nuss is capable of controlling the hardware. nuss has an upper > interface which is switchdev, and a lower interface which somehow acts > on the hardware, maybe invoking RPCs into the firmware? There are no RPCs used by the "am65-cpsw-nuss.c" driver. It assumes that it is the only user of CPSW Ethernet Switch. It doesn't interface with any firmware. Based on the switchdev framework, it receives commands from userspace which it then uses to directly write to CPSW's registers. > > So it is not too big a step to put the server functionality in Linux, > on top of the nuss driver. Maybe it isn't a big step but it doesn't help with the use-case that I have described above. For that reason, while it might be a "good to have" feature, it is not solving the problem. > > Now take a step backwards. This concept of a switch with different > CPUs attached to it is nothing special. > > Some Marvell switches have a Z80 connected to a dedicated port of the > switch. You can run applications on that Z80. Those applications might > be as simple as spanning tree, so you can have a white box 8 port > ethernet switch without needing an external CPU. But there is an SDK, > you could run any sort of application on the Z80. > > The microchip LAN996x switch has a Cortex A7 CPU, but also a PCIe > interface. Linux can control the switch via the PCIe interface, but > there could also be applications running on the Cortex. > > Look at the Broadcom BCM89586M: > https://docs.broadcom.com/doc/89586M-PB > > It is similar to the microchip device, an M7 CPU, and a PCIe > interface. It seems like a Linux host could be controlling the switch > via PCIe, and applications running on the M7. > > I expect there are more examples, but you get the idea. I have gone through the examples above. All of them are referring to the Hardware Capabilities of the Ethernet Switch, which aren't applicable to the CPSW Ethernet Switch. I am listing why each of them isn't applicable: 1. Marvel Z80 Switch: I assume that you are referring to: https://wiki.espressobin.net/tiki-index.php?page=Topaz+Switch with the "Integrated 200MHz Z80 microprocessor". CPSW doesn't have an embedded microprocessor dedicated to programming it. The closest it could get to the Z80 is the external R5 Core running EthFw as far as configuring the Switch is concerned. But how does it handle the use-case where there are applications running simultaneously on different cores, all of which require Ethernet Functionality with the same Ethernet Switch, in a dynamic manner? 2. Microchip LAN996x: CPSW doesn't have a PCIe interface. 3. Broadcom BCM89586M: Again, CPSW doesn't have a PCIe interface. An important point to note is that all applications you have mentioned are running on a single core. The current framework being proposed to solve the problem is for the use-case where there are applications running across various cores with different criticality (not all applications may be running all the time, Linux for example will be suspended as well). > > A completely different angle to look at is VF/PF and eswitches. A > server style CPU running virtual machines, a VM getting a VF passed > through to it. This is not something i know much about, so we might > need to pull in some data centre specialists. But we have a different > CPU, a virtual CPU, making use of part of the switch. Its the same > concept really. CPSW doesn't support SR-IOV. However, if you are referring to modelling CPSW as an SR-IOV capable Ethernet Switch by having EthFw pose as the Driver for the "Virtual" Physical Function of CPSW, with each Client Driver mapping to one of the modelled "Virtual" Virtual Functions exposed by EthFw, then yes, I will spend time looking at how that could be implemented. The term "Virtual" has been added in the previous sentence to clarify that CPSW isn't truly SR-IOV capable and we are simply making it look that way via EthFw. Even in SR-IOV, the communicatoin between PF and VF drivers happens via Hardware Mailbox which means RPMsg is coming back into the picture. The current implementation also is using RPMsg to exchange control information between EthFw and all the Clients. > > My main point is, please start with an abstract view of the problem. A > lot of the solution should be generic, it can be applied to all these > devices. And then there probably needs to be a small part which is > specific to TI devices. It could be parts of the solutions already > exist, e.g. VF/PF have port represents, which might be a useful > concept here as well. Switchdev exists and provides a generic > interface for configuring switches... The summary of the problem statement is: We require a framework in which the Ethernet Switch (CPSW) has to be shared across the applications running on different cores of the SoC. Since CPSW doesn't have an Integrated Processor, some core on the SoC has to act as the Central Entity which is responsible for arbitrating configuration requests from various cores and performing the appropriate configuration of CPSW. Additionally, apart from performing configuration of CPSW, the Central Entity is also responsible for allocating resources to different cores, including DMA Channels/Flows (There are 8 TX DMA Channels which have to be split across different cores to allow each core to send traffic to CPSW for example). CPSW should be functional even when some of the Clients (including Linux) might be suspended for Low Power use-case. The forwarding path of CPSW should be functional within 100s of milliseconds after the Bootloader stage. Kindly share your feedback on possible implementations to address the problem summarized above. Thank you for sharing your valuable feedback so far on this series. Regards, Siddharth.
On Sun, Jun 02, 2024 at 09:36:05AM +0530, Siddharth Vadapalli wrote: > On Sun, May 19, 2024 at 05:31:16PM +0200, Andrew Lunn wrote: > > Andrew, > > I have spent time going through your feedback, trying to understand your > suggestions. This email is the complete reply corresponding to my earlier > reply at: > https://lore.kernel.org/r/0b0c1b07-756e-439e-bfc5-53824fd2a61c@ti.com/ > which was simply meant to serve as an acknowledgement that I have seen > your email. > > > On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote: > > > The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on > > > a remote core to enable Ethernet functionality for applications running > > > on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW > > > Ethernet Switch on the SoC and acts as the Server, offering services to > > > Clients running on various cores. > > > > I'm not sure we as a community what this architecture. We want Linux > > to be driving the hardware, not firmware. So expect linux to be > > running the server. > > Due to the use-case requirements, Linux cannot be the server. Some of > the requirements are: > 1. Fast startup and configuration of CPSW independent of Linux and Other > OS running on any of the cores on the SoC. The configuration of CPSW has > to be performed in parallel while the Bootloader starts Linux. > 2. CPSW has to be functional and configurable even when Linux has been > suspended. One of the non-Linux Clients happens to be the AUTOSAR Client > which continues exchanging network data via CPSW even when Linux has > been suspended. So the server has to be functional even then, in order > to cater to the AUTOSAR Client's requests to configure CPSW. CPSW's > configuration is not static in the sense that it gets programmed and > will no longer be modified. Therefore the server has to be functional at > all times to update CPSW's configuration based on the demands of any of > the Clients. > > For more details about the Ethernet Switch Firmware (EthFw) and the set > of Clients running on remote cores, please refer: > https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/09_02_00_05/exports/docs/ethfw/docs/user_guide/ethfw_c_ug_top.html#ethfw_remote_clients Thanks for the links etc. I also admit, i did replied too soon, and should of read more of the patches. In Linux, we have two models with respect to switches. 1) They are external. Linux does not interact with them, other than sending them packets, and receiving packets from them. The switch might have some management interface, SNMP, HTTP, etc, but it is not linuxs job to manage the switch. Linux just has its NIC connected to the port of switch using a cable. This is the model used for a very long time. 2) The Linux kernel is controlling the switch, configuration is performed from userspace using iproute2. Switchdev is used internally to interface between the linux network stack and the switch driver. Depending on implementation, linux can either directly write switch registers, or it can perform an RPC to firmware running on the switch. But this is an implementation detail, Linux is in control of all the ports, all the routing/switching, IGMP snooping, STP, PTP, etc. Could what Linux sees of this hardware fit into the first model? Linux just sees a bunch of NICs connected to a switch? The switch is remote, linux has no control over it. Linux acts purely as a client for low level protocols like PTP, IGMP snooping, etc. It has no knowledge of other ports of the switch, there up/down state, what STP is doing in the switch, how PTP is forwarding packets from the upstream port to the downstream ports. Linux has no idea and no access to the address lookup engines in the switch. The switch is colocated in the same silicon, but all linux has is some ports connected to the switch, nothing more? What is interesting is Realteks current driver work for there automotive system. There CPU has one MAC which is connected to the internal switch. But they have a similar setup, Linux is not controlling the switch, some other firmware is. They have PTP, IGMP snooping, STP etc running in firmware. Linux just has a NIC connected to the switch as an end system. If you do want to add a third model, where Linux has some insight into the switch, you need to coordinate with other vendors in the automotive world, and come up with a model which everybody can use. What i don't want is a TI model, followed by a Realtek model, followed by a vendor XYZ model. So if you need more than what the first model above provides, you will need to get a consortium of vendors together to design a new model a few vendors agree on. Andrew
On Tue, Jun 04, 2024 at 04:20:16PM +0200, Andrew Lunn wrote: > On Sun, Jun 02, 2024 at 09:36:05AM +0530, Siddharth Vadapalli wrote: [...] > > If you do want to add a third model, where Linux has some insight into > the switch, you need to coordinate with other vendors in the > automotive world, and come up with a model which everybody can > use. What i don't want is a TI model, followed by a Realtek model, > followed by a vendor XYZ model. So if you need more than what the > first model above provides, you will need to get a consortium of > vendors together to design a new model a few vendors agree on. I believe that a third model is required given the System Architecture and the use-case that it must cater to. I agree completely that having a vendor specific implementation should always be the last step when it is just not possible to generalize any portion of the implementation. I will describe the existing Architecture on the TI SoC and will also attempt to generalize the implementation below. I hope that you could review it and guide me towards the generic, vendor-agnostic implementation which will also address the use-case that this series corresponds to. I am willing to work on the generic implementation since I assume that this series does keep it generic enough that it could be extended to be vendor independent. So there might be minor changes required when switching to the generic model. On the other hand, based on the description that I provide below, if you think that the existing models can also be slightly modified to accomodate the use-case, I will surely take that into consideration and work on the corresponding implementation. System Architecture and Implementation Details ============================================== The CPSW Ethernet Switch has a single Host Port (CPU facing port) through which it can receive data from the Host(s) and transmit data to the Host(s). The exchange of data occurs via TX/RX DMA Channels (Hardware Queues). These Hardware Queues are a limited resource (8 TX Channels and up to 64 RX Flows). If the Operating System on any of the cores is the sole user of CPSW then all of these Hardware Queues can be claimed by that OS. However, when CPSW has to be shared across the Operating Systems on various cores with the aim of enabling Ethernet Functionality for the Applications running on different cores, it is necessary to share these Hardware Queues in a manner that prevents conflicts. On the control path which corresponds to the configuration of CPSW to get it up and running, since there is no Integrated Processor within CPSW that can be programmed with a startup configuration, either the Operating System or Firmware running on one of the cores has to take the responsibility of setting it. One option in this case happens to be the Ethernet Switch Firmware (EthFw) which is loaded by the Bootloader on a remote core at the same time that Linux and other Operating Systems begin booting. EthFw quickly powers on and configures CPSW getting the Forwarding Path functional. Once Linux and other Operating Systems on various cores are ready, they can communicate with EthFw to obtain details of the Hardware Queues allocated to them to exchange data with CPSW. With the knowledge of the Hardware Queues that have been allocated, Linux can use the DMA APIs to setup these queues to exchange data with CPSW. Setting up the Hardware Queues alone isn't sufficient to exchange data with the external network. Consider the following example: The ethX interface in userspace which has been created to transmit/receive data to/from CPSW has the user-assigned MAC Address of "M". The ping command is run with the destination IP of "D". This results in an ARP request sent from ethX which is transmitted out of all MAC Ports of CPSW since it is a Broadcast request. Assuming that "D" is a valid destination IP, the ARP reply is received on one of the MAC Ports which is now a Unicast reply with the destination MAC Address of "M". The ALE (Address Lookup Engine) in CPSW has learnt that the MAC Address "M" corresponds to the Host Port when the ARP request was sent out. So the Unicast reply isn't dropped. The challenge however is determining which RX DMA Channel (Flow) to send the Unicast reply on. In the case of a single Operating System owning all Hardware Queues, sending it on any of the RX DMA Channels would have worked. In the current case where the RX DMA Channels map to different Hosts (Operating Systems and Applications), the mapping between the MAC Address "M" and the RX DMA Channel has to be setup to ensure that the correct Host receives the ARP reply. This necessitates a method to inform the MAC Address "M" associated with the interface ethX to EthFw so that EthFw can setup the MAC Address "M" to RX DMA Channel map accordingly. At this point, Linux can exchange data with the external network via CPSW, but no device on the external network can initiate the communication by itself unless it already has the ARP entry for the IP Address of ethX. That's because CPSW doesn't support packet replication implying that any Broadcast/Multicast packets received on the MAC Ports can only be sent on one of the RX DMA Channels. So the Broadcast/Multicast packets can only be received by one Host. Consider the following example: A PC on the network tries to ping the IP Address of ethX. In both of the following cases: 1. Linux hasn't yet exchanged data with the PC via ethX. 2. The MAC Address of ethX has changed. the PC sends an ARP request to one of the MAC Ports on CPSW to figure out the MAC Address of ethX. Since the ARP request is a Broadcast request, it is not possible for CPSW to determine the correct Host, since the Broadcast MAC isn't unique to any Host. So CPSW is forced to send the Broadcast request to a preconfigured RX DMA Channel which in this case happens to be the one mapped to EthFw. Thus, if EthFw is aware of the IP Address of ethX, it can generate and send the ARP reply containing the MAC Address "M" of ethX that it was informed of. With this, the PC can initiate communication with Linux as well. Similarly, in the case of Multicast packets, if Linux wishes to receive certain Multicast packets, it needs to inform the same to EthFw which shall then replicate the Multicast packets it received from CPSW and transmit them via alternate means (Shared Memory for example) to Linux. The following is a summary of the steps so far required to enable Ethernet Functionality for applications running on Linux: 1. Determine and setup the Hardware Queues allocated to Linux 2. Inform the MAC Address of ethX to EthFw 3. Inform the IP Address of ethX to EthFw 4. Inform any of the Multicast Addresses associated with ethX to EthFw All data between Linux (Or any Operating System) and EthFw is exchanged via the Hardware Mailboxes with the help of the RPMsg framework. Since all the resource allocation information comes from EthFw, the vendor-specific implementation in the Linux Client is limited to the DMA APIs used to setup the Hardware Queues and to transmit/receive data with the Ethernet Switch. Therefore, it might be possible to move most of the vendor specific implementation to the Switch Configuration Firmware (similar to EthFw), to make the Linux Client implementation as generic and vendor agnostic as possible. I believe that this series more or less does the same, just using custom terminology which can be made generic. I can update this series to a generic implementation along with proper documentation and naming convention to enable any vendor to reuse the same without having to modify the implementation. The RPMsg ABIs can be given generic names with extensive documentation and also designed to be extensible enough to cater to functional enhancements over time. Kindly let me know your thoughts on this. Regards, Siddharth.
diff --git a/Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst b/Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst new file mode 100644 index 000000000000..46bff67b3446 --- /dev/null +++ b/Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst @@ -0,0 +1,182 @@ +.. SPDX-License-Identifier: GPL-2.0-only or MIT + +========================================== +Texas Instruments CPSW Proxy Client driver +========================================== + +Introduction +============ + +The CPSW (Common Platform Switch) Ethernet Switch on TI's K3 SoCs provides +Ethernet functionality. There may be multiple instances of CPSW on a single +SoC. The term "CPSWnG" is used to indicate the number of MAC Ports supported +by a specific instance of CPSW. CPSWnG indicates that the peripheral has +(n-1) MAC Ports and 1 Host Port. Examples of existing instances are: +CPSW2G => 1 MAC Port and 1 Host Port +CPSW3G => 2 MAC Ports and 1 Host Port +CPSW5G => 4 MAC Ports and 1 Host Port +CPSW9G => 8 MAC Ports and 1 Host Port + +The presence of 2 or more MAC Ports implies that Hardware Switching can +be enabled between the MAC Ports if required. + +The "am65-cpsw-nuss.c" driver in Linux at: +drivers/net/ethernet/ti/am65-cpsw-nuss.c +provides Ethernet functionality for applications on Linux. +It also handles both the control-path and data-path, namely: +Control => Configuration of the CPSW Peripheral +Data => Configuration of the DMA Channels to transmit/receive data + +The aforementioned configuration supports use-cases where all applications +which require Ethernet functionality are only running on Linux. + +However, there are use-cases where applications running on different +Operating Systems across multiple cores on the SoC require Ethernet +functionality. Such use-cases can be supported by implementing a +Client-Server model to share the data-path among Clients while the Server +owns the control-path. + +On TI's K3 SoCs (J721E, J7200 and J784S4 in particular), the Ethernet Switch +Firmware (EthFw) running on the MAIN R5F core acts as the Server and +configures the CPSWnG instance (CPSW5G on J7200 and CPSW9G on J721E, J784S4) +of the CPSW Ethernet Switch on the SoC. The Clients running on various cores +communicate with EthFw via RPMsg (Remote Processor Messaging) to request +resource allocation details during initialization, followed by requesting +EthFw to enable features supported by CPSW based on the features required +by the applications running on the respective cores. + +EthFw handles requests from the Clients and evaluates them before configuring +CPSW based on the request. Since no Client is actually in control of CPSW and +only requests EthFw for configuring CPSW, EthFw acts as the proxy for the +Clients. Thus, the Linux Client which interfaces with EthFw is named: +CPSW Proxy Client + +The data-path for the CPSW Proxy Client driver remains identical to the +"am65-cpsw-nuss.c" driver which happens to be DMA. It is only the control-path +that is different. + +Client-Server discovery occurs over the RPMsg-Bus. EthFw announces its +RPMsg Endpoint name over the RPMsg-Bus. The CPSW Proxy Client driver +registers itself with the Linux RPMsg framework to be probed for the same +Endpoint name. Following probe, the Linux Client driver begins communicating +with EthFw and queries details of the resources available for the Linux Client. + +Terminology +=========== + +Virtual Port + A Virtual Port refers to the Software View of an Ethernet MAC Port. + There are two types of Virtual Ports: + 1. Virtual MAC Only Port + 2. Virtual Switch Port + +Virtual MAC Only Port + A Virtual MAC only Port refers to a dedicated physical MAC Port for + a Client. This corresponds to MAC Mode of operation in Ethernet + Terminology. All traffic sent to or received from the Physical + MAC Port is that of the Client to which the Virtual MAC Only Port + has been allocated. + +Virtual Switch Port + A Virtual Switch Port refers to a group of physical MAC ports with + Switching enabled across them. This implies that any traffic sent + to the Port from a Client could potentially exit a Physical MAC + Port along with the traffic from other Clients. Similarly, the traffic + received on the Port by a Client could have potentially ingressed + on a Physical MAC Port along with the traffic meant for other Clients. + While the ALE (Address Lookup Engine) handles segregating the traffic, + and the CPSW Ethernet Switch places traffic on dedicated RX DMA Flows + meant for a single Client, it is worth noting that the bandwidths + of the Physical MAC Port are shared by Clients when traffic is sent to + or received from a Virtual Switch Port. + +Network Interface + The user-visible interface in Linux userspace exposed to applications + that serves as the entry/exit point for traffic to/from the Virtual + Ports. A single network interface (ethX) maps to either a Virtual + MAC Only Port or a Virtual Switch Port. + +C2S + RPMsg source is Client and destination is Server. + +S2C + RPMsg source is Server and destination is Client. + +Initialization Sequence +======================= + +The sequence of message exchanges between the Client driver and EthFw starting +from the driver probe and ending with the interfaces being brought up is as +follows: +1. C2S ETHFW_VIRT_PORT_INFO requesting details of Virtual Ports available + for the Linux Client. +2. S2C response containing requested details +3. C2S ETHFW_VIRT_PORT_ATTACH request for each Virtual Port allocated during + step 2. +4. S2C response containing details of the MTU Size, number of Tx DMA Channels + and RX DMA Flows for the specified Virtual Port. The *Features* associated + with the Virtual Port are also shared such as Multicast Filtering capability. +5. C2S ETHFW_ALLOC_RX request for each RX DMA Flow for a Virtual Port. +6. S2C response containing details of the RX PSI-L Thread ID, Flow base and + Flow offset. +7. C2S ETHFW_ALLOC_TX request for each TX DMA Channel for a Virtual Port. +8. S2C response containing details of the TX PSI-L Thread ID. +9. C2S ETHFW_ALLOC_MAC request for each Virtual Port. +10. S2C response containing the MAC Address corresponding to the Virtual Port. +11. C2S ETHFW_MAC_REGISTER request for each Virtual Port with the MAC Address + allocated in step 10. This is necessary to steer packets that ingress on + the MAC Ports of CPSW onto the RX DMA Flow for the Virtual Port in order + to allow the Client to receive the packets. +12. S2C response indicating status of request. +13. C2S ETHFW_IPv4_REGISTER request *only* for Virtual Switch Port interface. + The IPv4 address assigned to the "ethX" network interface in Linux + corresponding to the Virtual Switch Port interface has to be registered + with EthFw. This is due to the reason that all Broadcast requests including + ARP requests received by the MAC Ports corresponding to the Virtual Switch + Port are consumed solely be EthFw. Such traffic is sent to Clients by + alternate methods. Therefore EthFw needs to know the IPv4 address for the + "ethX" network interface in Linux in order to automatically respond to + ARP requests, thereby enabling Unicast communication. +14. S2C response indicating status of request. +15. C2S ETHFW_MCAST_FILTER_ADD request to register the Multicast Addresses + associated with the network interface corresponding to the Virtual Port + which has the Multicast Filtering capability. +16. S2C response indicating status of request. +17. C2S ETHFW_MCAST_FILTER_DEL request to deregister the Multicast Addresses + associated with the network interface corresponding to the Virtual Port + which has the Multicast Filtering capability. +18. S2C response indicating status of request. + +Shutdown Sequence +================= + +The sequence of message exchanges between the Client driver and EthFw on module +removal are as follows: +1. C2S ETHFW_MAC_DEREGISTER request to deregister the MAC Address for each + Virtual Port. +2. S2C response indicating status of request. +3. C2S ETHFW_MCAST_FILTER_DEL request to deregister the Multicast Addresses + associated with the network interface corresponding to the Virtual Port + which has the Multicast Filtering capability. +4. S2C response indicating status of request. +5. C2S ETHFW_FREE_MAC request to release the MAC Address allocated to each + Virtual Port. +6. S2C response indicating status of request. +7. C2S ETHFW_FREE_TX request to release the TX DMA Channel for each TX Channel + for every Virtual Port. +8. S2C response indicating status of request. +9. C2S ETHFW_FREE_RX request to release the RX DMA Flow for each RX Channel + for every Virtual Port. +10. S2C response indicating status of request. +11. C2S ETHFW_VIRT_PORT_DETACH request to release each Virtual Port. +12. S2C response indicating status of request. + +Features Supported +================== + +The set of features supported in addition to providing basic Ethernet +Functionality are: +1. Multicast Filtering +2. Determining Link Status of the network interface corresponding to the + Virtual MAC Only port via ethtool. +3. Interrupt Pacing/Coalescing
The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on a remote core to enable Ethernet functionality for applications running on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW Ethernet Switch on the SoC and acts as the Server, offering services to Clients running on various cores. Signed-off-by: Siddharth Vadapalli <s-vadapalli@ti.com> --- .../ethernet/ti/cpsw_proxy_client.rst | 182 ++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst