| Message ID | 20240518124234.2671651-2-s-vadapalli@ti.com (mailing list archive) |
|---|---|
| State | RFC |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | Add CPSW Proxy Client driver | expand |
On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote: > The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on > a remote core to enable Ethernet functionality for applications running > on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW > Ethernet Switch on the SoC and acts as the Server, offering services to > Clients running on various cores. I'm not sure we as a community what this architecture. We want Linux to be driving the hardware, not firmware. So expect linux to be running the server. > +The "am65-cpsw-nuss.c" driver in Linux at: > +drivers/net/ethernet/ti/am65-cpsw-nuss.c > +provides Ethernet functionality for applications on Linux. > +It also handles both the control-path and data-path, namely: > +Control => Configuration of the CPSW Peripheral > +Data => Configuration of the DMA Channels to transmit/receive data So nuss is capable of controlling the hardware. nuss has an upper interface which is switchdev, and a lower interface which somehow acts on the hardware, maybe invoking RPCs into the firmware? So it is not too big a step to put the server functionality in Linux, on top of the nuss driver. Now take a step backwards. This concept of a switch with different CPUs attached to it is nothing special. Some Marvell switches have a Z80 connected to a dedicated port of the switch. You can run applications on that Z80. Those applications might be as simple as spanning tree, so you can have a white box 8 port ethernet switch without needing an external CPU. But there is an SDK, you could run any sort of application on the Z80. The microchip LAN996x switch has a Cortex A7 CPU, but also a PCIe interface. Linux can control the switch via the PCIe interface, but there could also be applications running on the Cortex. Look at the Broadcom BCM89586M: https://docs.broadcom.com/doc/89586M-PB It is similar to the microchip device, an M7 CPU, and a PCIe interface. It seems like a Linux host could be controlling the switch via PCIe, and applications running on the M7. I expect there are more examples, but you get the idea. A completely different angle to look at is VF/PF and eswitches. A server style CPU running virtual machines, a VM getting a VF passed through to it. This is not something i know much about, so we might need to pull in some data centre specialists. But we have a different CPU, a virtual CPU, making use of part of the switch. Its the same concept really. My main point is, please start with an abstract view of the problem. A lot of the solution should be generic, it can be applied to all these devices. And then there probably needs to be a small part which is specific to TI devices. It could be parts of the solutions already exist, e.g. VF/PF have port represents, which might be a useful concept here as well. Switchdev exists and provides a generic interface for configuring switches... Andrew
On Sun, May 19, 2024 at 05:31:16PM +0200, Andrew Lunn wrote: > On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote: > > The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on > > a remote core to enable Ethernet functionality for applications running > > on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW > > Ethernet Switch on the SoC and acts as the Server, offering services to > > Clients running on various cores. > > I'm not sure we as a community what this architecture. We want Linux > to be driving the hardware, not firmware. So expect linux to be > running the server. > > > +The "am65-cpsw-nuss.c" driver in Linux at: > > +drivers/net/ethernet/ti/am65-cpsw-nuss.c > > +provides Ethernet functionality for applications on Linux. > > +It also handles both the control-path and data-path, namely: > > +Control => Configuration of the CPSW Peripheral > > +Data => Configuration of the DMA Channels to transmit/receive data > > So nuss is capable of controlling the hardware. nuss has an upper > interface which is switchdev, and a lower interface which somehow acts > on the hardware, maybe invoking RPCs into the firmware? > > So it is not too big a step to put the server functionality in Linux, > on top of the nuss driver. Andrew, Thank you for reviewing the patch and sharing your feedback. While I have come across other Switch Designs / Architecture, I am yet to go through the one you have mentioned below. I will go through it in detail and will follow up with my understanding in a future reply. This reply is intended to be an acknowledgment that I have read your feedback. I also wanted to clarify the use-case which this series targets. The requirements of the use-case are: 1. Independent Ethernet Switch functionality: Switch operation and configuration when Linux is not functional (Fast startup, Low Power Mode, Safety use-cases). 2. Dynamic Ethernet Switch configuration changes performed based on the applications which run on various cores. [...] Regards, Siddharth.
> Andrew, > > Thank you for reviewing the patch and sharing your feedback. While I > have come across other Switch Designs / Architecture, I am yet to go > through the one you have mentioned below. I will go through it in detail > and will follow up with my understanding in a future reply. This reply > is intended to be an acknowledgment that I have read your feedback. > I also wanted to clarify the use-case which this series targets. The > requirements of the use-case are: > 1. Independent Ethernet Switch functionality: Switch operation and > configuration when Linux is not functional (Fast startup, Low Power > Mode, Safety use-cases). > 2. Dynamic Ethernet Switch configuration changes performed based on the > applications which run on various cores. Please make sure these requirements are clearly stated in the design. The support for switches in Linux has initially come from big data centre switches, and smaller SOHO switches you found in OpenWRT class devices. The switchdev model has worked well so far for these use cases. However, i do understand you have additional requirements. Ideally we want to extend the existing model to support additional use cases, not create a second parallel model. And we want a vendor agnostic extensions of the switchdev model, something which all automotive vendors can use, and non-automotive systems which have a similar architecture. Andrew
On Sun, May 19, 2024 at 05:31:16PM +0200, Andrew Lunn wrote: Andrew, I have spent time going through your feedback, trying to understand your suggestions. This email is the complete reply corresponding to my earlier reply at: https://lore.kernel.org/r/0b0c1b07-756e-439e-bfc5-53824fd2a61c@ti.com/ which was simply meant to serve as an acknowledgement that I have seen your email. > On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote: > > The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on > > a remote core to enable Ethernet functionality for applications running > > on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW > > Ethernet Switch on the SoC and acts as the Server, offering services to > > Clients running on various cores. > > I'm not sure we as a community what this architecture. We want Linux > to be driving the hardware, not firmware. So expect linux to be > running the server. Due to the use-case requirements, Linux cannot be the server. Some of the requirements are: 1. Fast startup and configuration of CPSW independent of Linux and Other OS running on any of the cores on the SoC. The configuration of CPSW has to be performed in parallel while the Bootloader starts Linux. 2. CPSW has to be functional and configurable even when Linux has been suspended. One of the non-Linux Clients happens to be the AUTOSAR Client which continues exchanging network data via CPSW even when Linux has been suspended. So the server has to be functional even then, in order to cater to the AUTOSAR Client's requests to configure CPSW. CPSW's configuration is not static in the sense that it gets programmed and will no longer be modified. Therefore the server has to be functional at all times to update CPSW's configuration based on the demands of any of the Clients. For more details about the Ethernet Switch Firmware (EthFw) and the set of Clients running on remote cores, please refer: https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/09_02_00_05/exports/docs/ethfw/docs/user_guide/ethfw_c_ug_top.html#ethfw_remote_clients > > > +The "am65-cpsw-nuss.c" driver in Linux at: > > +drivers/net/ethernet/ti/am65-cpsw-nuss.c > > +provides Ethernet functionality for applications on Linux. > > +It also handles both the control-path and data-path, namely: > > +Control => Configuration of the CPSW Peripheral > > +Data => Configuration of the DMA Channels to transmit/receive data > > So nuss is capable of controlling the hardware. nuss has an upper > interface which is switchdev, and a lower interface which somehow acts > on the hardware, maybe invoking RPCs into the firmware? There are no RPCs used by the "am65-cpsw-nuss.c" driver. It assumes that it is the only user of CPSW Ethernet Switch. It doesn't interface with any firmware. Based on the switchdev framework, it receives commands from userspace which it then uses to directly write to CPSW's registers. > > So it is not too big a step to put the server functionality in Linux, > on top of the nuss driver. Maybe it isn't a big step but it doesn't help with the use-case that I have described above. For that reason, while it might be a "good to have" feature, it is not solving the problem. > > Now take a step backwards. This concept of a switch with different > CPUs attached to it is nothing special. > > Some Marvell switches have a Z80 connected to a dedicated port of the > switch. You can run applications on that Z80. Those applications might > be as simple as spanning tree, so you can have a white box 8 port > ethernet switch without needing an external CPU. But there is an SDK, > you could run any sort of application on the Z80. > > The microchip LAN996x switch has a Cortex A7 CPU, but also a PCIe > interface. Linux can control the switch via the PCIe interface, but > there could also be applications running on the Cortex. > > Look at the Broadcom BCM89586M: > https://docs.broadcom.com/doc/89586M-PB > > It is similar to the microchip device, an M7 CPU, and a PCIe > interface. It seems like a Linux host could be controlling the switch > via PCIe, and applications running on the M7. > > I expect there are more examples, but you get the idea. I have gone through the examples above. All of them are referring to the Hardware Capabilities of the Ethernet Switch, which aren't applicable to the CPSW Ethernet Switch. I am listing why each of them isn't applicable: 1. Marvel Z80 Switch: I assume that you are referring to: https://wiki.espressobin.net/tiki-index.php?page=Topaz+Switch with the "Integrated 200MHz Z80 microprocessor". CPSW doesn't have an embedded microprocessor dedicated to programming it. The closest it could get to the Z80 is the external R5 Core running EthFw as far as configuring the Switch is concerned. But how does it handle the use-case where there are applications running simultaneously on different cores, all of which require Ethernet Functionality with the same Ethernet Switch, in a dynamic manner? 2. Microchip LAN996x: CPSW doesn't have a PCIe interface. 3. Broadcom BCM89586M: Again, CPSW doesn't have a PCIe interface. An important point to note is that all applications you have mentioned are running on a single core. The current framework being proposed to solve the problem is for the use-case where there are applications running across various cores with different criticality (not all applications may be running all the time, Linux for example will be suspended as well). > > A completely different angle to look at is VF/PF and eswitches. A > server style CPU running virtual machines, a VM getting a VF passed > through to it. This is not something i know much about, so we might > need to pull in some data centre specialists. But we have a different > CPU, a virtual CPU, making use of part of the switch. Its the same > concept really. CPSW doesn't support SR-IOV. However, if you are referring to modelling CPSW as an SR-IOV capable Ethernet Switch by having EthFw pose as the Driver for the "Virtual" Physical Function of CPSW, with each Client Driver mapping to one of the modelled "Virtual" Virtual Functions exposed by EthFw, then yes, I will spend time looking at how that could be implemented. The term "Virtual" has been added in the previous sentence to clarify that CPSW isn't truly SR-IOV capable and we are simply making it look that way via EthFw. Even in SR-IOV, the communicatoin between PF and VF drivers happens via Hardware Mailbox which means RPMsg is coming back into the picture. The current implementation also is using RPMsg to exchange control information between EthFw and all the Clients. > > My main point is, please start with an abstract view of the problem. A > lot of the solution should be generic, it can be applied to all these > devices. And then there probably needs to be a small part which is > specific to TI devices. It could be parts of the solutions already > exist, e.g. VF/PF have port represents, which might be a useful > concept here as well. Switchdev exists and provides a generic > interface for configuring switches... The summary of the problem statement is: We require a framework in which the Ethernet Switch (CPSW) has to be shared across the applications running on different cores of the SoC. Since CPSW doesn't have an Integrated Processor, some core on the SoC has to act as the Central Entity which is responsible for arbitrating configuration requests from various cores and performing the appropriate configuration of CPSW. Additionally, apart from performing configuration of CPSW, the Central Entity is also responsible for allocating resources to different cores, including DMA Channels/Flows (There are 8 TX DMA Channels which have to be split across different cores to allow each core to send traffic to CPSW for example). CPSW should be functional even when some of the Clients (including Linux) might be suspended for Low Power use-case. The forwarding path of CPSW should be functional within 100s of milliseconds after the Bootloader stage. Kindly share your feedback on possible implementations to address the problem summarized above. Thank you for sharing your valuable feedback so far on this series. Regards, Siddharth.
On Sun, Jun 02, 2024 at 09:36:05AM +0530, Siddharth Vadapalli wrote: > On Sun, May 19, 2024 at 05:31:16PM +0200, Andrew Lunn wrote: > > Andrew, > > I have spent time going through your feedback, trying to understand your > suggestions. This email is the complete reply corresponding to my earlier > reply at: > https://lore.kernel.org/r/0b0c1b07-756e-439e-bfc5-53824fd2a61c@ti.com/ > which was simply meant to serve as an acknowledgement that I have seen > your email. > > > On Sat, May 18, 2024 at 06:12:07PM +0530, Siddharth Vadapalli wrote: > > > The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on > > > a remote core to enable Ethernet functionality for applications running > > > on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW > > > Ethernet Switch on the SoC and acts as the Server, offering services to > > > Clients running on various cores. > > > > I'm not sure we as a community what this architecture. We want Linux > > to be driving the hardware, not firmware. So expect linux to be > > running the server. > > Due to the use-case requirements, Linux cannot be the server. Some of > the requirements are: > 1. Fast startup and configuration of CPSW independent of Linux and Other > OS running on any of the cores on the SoC. The configuration of CPSW has > to be performed in parallel while the Bootloader starts Linux. > 2. CPSW has to be functional and configurable even when Linux has been > suspended. One of the non-Linux Clients happens to be the AUTOSAR Client > which continues exchanging network data via CPSW even when Linux has > been suspended. So the server has to be functional even then, in order > to cater to the AUTOSAR Client's requests to configure CPSW. CPSW's > configuration is not static in the sense that it gets programmed and > will no longer be modified. Therefore the server has to be functional at > all times to update CPSW's configuration based on the demands of any of > the Clients. > > For more details about the Ethernet Switch Firmware (EthFw) and the set > of Clients running on remote cores, please refer: > https://software-dl.ti.com/jacinto7/esd/processor-sdk-rtos-jacinto7/09_02_00_05/exports/docs/ethfw/docs/user_guide/ethfw_c_ug_top.html#ethfw_remote_clients Thanks for the links etc. I also admit, i did replied too soon, and should of read more of the patches. In Linux, we have two models with respect to switches. 1) They are external. Linux does not interact with them, other than sending them packets, and receiving packets from them. The switch might have some management interface, SNMP, HTTP, etc, but it is not linuxs job to manage the switch. Linux just has its NIC connected to the port of switch using a cable. This is the model used for a very long time. 2) The Linux kernel is controlling the switch, configuration is performed from userspace using iproute2. Switchdev is used internally to interface between the linux network stack and the switch driver. Depending on implementation, linux can either directly write switch registers, or it can perform an RPC to firmware running on the switch. But this is an implementation detail, Linux is in control of all the ports, all the routing/switching, IGMP snooping, STP, PTP, etc. Could what Linux sees of this hardware fit into the first model? Linux just sees a bunch of NICs connected to a switch? The switch is remote, linux has no control over it. Linux acts purely as a client for low level protocols like PTP, IGMP snooping, etc. It has no knowledge of other ports of the switch, there up/down state, what STP is doing in the switch, how PTP is forwarding packets from the upstream port to the downstream ports. Linux has no idea and no access to the address lookup engines in the switch. The switch is colocated in the same silicon, but all linux has is some ports connected to the switch, nothing more? What is interesting is Realteks current driver work for there automotive system. There CPU has one MAC which is connected to the internal switch. But they have a similar setup, Linux is not controlling the switch, some other firmware is. They have PTP, IGMP snooping, STP etc running in firmware. Linux just has a NIC connected to the switch as an end system. If you do want to add a third model, where Linux has some insight into the switch, you need to coordinate with other vendors in the automotive world, and come up with a model which everybody can use. What i don't want is a TI model, followed by a Realtek model, followed by a vendor XYZ model. So if you need more than what the first model above provides, you will need to get a consortium of vendors together to design a new model a few vendors agree on. Andrew
On Tue, Jun 04, 2024 at 04:20:16PM +0200, Andrew Lunn wrote: > On Sun, Jun 02, 2024 at 09:36:05AM +0530, Siddharth Vadapalli wrote: [...] > > If you do want to add a third model, where Linux has some insight into > the switch, you need to coordinate with other vendors in the > automotive world, and come up with a model which everybody can > use. What i don't want is a TI model, followed by a Realtek model, > followed by a vendor XYZ model. So if you need more than what the > first model above provides, you will need to get a consortium of > vendors together to design a new model a few vendors agree on. I believe that a third model is required given the System Architecture and the use-case that it must cater to. I agree completely that having a vendor specific implementation should always be the last step when it is just not possible to generalize any portion of the implementation. I will describe the existing Architecture on the TI SoC and will also attempt to generalize the implementation below. I hope that you could review it and guide me towards the generic, vendor-agnostic implementation which will also address the use-case that this series corresponds to. I am willing to work on the generic implementation since I assume that this series does keep it generic enough that it could be extended to be vendor independent. So there might be minor changes required when switching to the generic model. On the other hand, based on the description that I provide below, if you think that the existing models can also be slightly modified to accomodate the use-case, I will surely take that into consideration and work on the corresponding implementation. System Architecture and Implementation Details ============================================== The CPSW Ethernet Switch has a single Host Port (CPU facing port) through which it can receive data from the Host(s) and transmit data to the Host(s). The exchange of data occurs via TX/RX DMA Channels (Hardware Queues). These Hardware Queues are a limited resource (8 TX Channels and up to 64 RX Flows). If the Operating System on any of the cores is the sole user of CPSW then all of these Hardware Queues can be claimed by that OS. However, when CPSW has to be shared across the Operating Systems on various cores with the aim of enabling Ethernet Functionality for the Applications running on different cores, it is necessary to share these Hardware Queues in a manner that prevents conflicts. On the control path which corresponds to the configuration of CPSW to get it up and running, since there is no Integrated Processor within CPSW that can be programmed with a startup configuration, either the Operating System or Firmware running on one of the cores has to take the responsibility of setting it. One option in this case happens to be the Ethernet Switch Firmware (EthFw) which is loaded by the Bootloader on a remote core at the same time that Linux and other Operating Systems begin booting. EthFw quickly powers on and configures CPSW getting the Forwarding Path functional. Once Linux and other Operating Systems on various cores are ready, they can communicate with EthFw to obtain details of the Hardware Queues allocated to them to exchange data with CPSW. With the knowledge of the Hardware Queues that have been allocated, Linux can use the DMA APIs to setup these queues to exchange data with CPSW. Setting up the Hardware Queues alone isn't sufficient to exchange data with the external network. Consider the following example: The ethX interface in userspace which has been created to transmit/receive data to/from CPSW has the user-assigned MAC Address of "M". The ping command is run with the destination IP of "D". This results in an ARP request sent from ethX which is transmitted out of all MAC Ports of CPSW since it is a Broadcast request. Assuming that "D" is a valid destination IP, the ARP reply is received on one of the MAC Ports which is now a Unicast reply with the destination MAC Address of "M". The ALE (Address Lookup Engine) in CPSW has learnt that the MAC Address "M" corresponds to the Host Port when the ARP request was sent out. So the Unicast reply isn't dropped. The challenge however is determining which RX DMA Channel (Flow) to send the Unicast reply on. In the case of a single Operating System owning all Hardware Queues, sending it on any of the RX DMA Channels would have worked. In the current case where the RX DMA Channels map to different Hosts (Operating Systems and Applications), the mapping between the MAC Address "M" and the RX DMA Channel has to be setup to ensure that the correct Host receives the ARP reply. This necessitates a method to inform the MAC Address "M" associated with the interface ethX to EthFw so that EthFw can setup the MAC Address "M" to RX DMA Channel map accordingly. At this point, Linux can exchange data with the external network via CPSW, but no device on the external network can initiate the communication by itself unless it already has the ARP entry for the IP Address of ethX. That's because CPSW doesn't support packet replication implying that any Broadcast/Multicast packets received on the MAC Ports can only be sent on one of the RX DMA Channels. So the Broadcast/Multicast packets can only be received by one Host. Consider the following example: A PC on the network tries to ping the IP Address of ethX. In both of the following cases: 1. Linux hasn't yet exchanged data with the PC via ethX. 2. The MAC Address of ethX has changed. the PC sends an ARP request to one of the MAC Ports on CPSW to figure out the MAC Address of ethX. Since the ARP request is a Broadcast request, it is not possible for CPSW to determine the correct Host, since the Broadcast MAC isn't unique to any Host. So CPSW is forced to send the Broadcast request to a preconfigured RX DMA Channel which in this case happens to be the one mapped to EthFw. Thus, if EthFw is aware of the IP Address of ethX, it can generate and send the ARP reply containing the MAC Address "M" of ethX that it was informed of. With this, the PC can initiate communication with Linux as well. Similarly, in the case of Multicast packets, if Linux wishes to receive certain Multicast packets, it needs to inform the same to EthFw which shall then replicate the Multicast packets it received from CPSW and transmit them via alternate means (Shared Memory for example) to Linux. The following is a summary of the steps so far required to enable Ethernet Functionality for applications running on Linux: 1. Determine and setup the Hardware Queues allocated to Linux 2. Inform the MAC Address of ethX to EthFw 3. Inform the IP Address of ethX to EthFw 4. Inform any of the Multicast Addresses associated with ethX to EthFw All data between Linux (Or any Operating System) and EthFw is exchanged via the Hardware Mailboxes with the help of the RPMsg framework. Since all the resource allocation information comes from EthFw, the vendor-specific implementation in the Linux Client is limited to the DMA APIs used to setup the Hardware Queues and to transmit/receive data with the Ethernet Switch. Therefore, it might be possible to move most of the vendor specific implementation to the Switch Configuration Firmware (similar to EthFw), to make the Linux Client implementation as generic and vendor agnostic as possible. I believe that this series more or less does the same, just using custom terminology which can be made generic. I can update this series to a generic implementation along with proper documentation and naming convention to enable any vendor to reuse the same without having to modify the implementation. The RPMsg ABIs can be given generic names with extensive documentation and also designed to be extensible enough to cater to functional enhancements over time. Kindly let me know your thoughts on this. Regards, Siddharth.
> System Architecture and Implementation Details > ============================================== > > The CPSW Ethernet Switch has a single Host Port (CPU facing port) through > which it can receive data from the Host(s) and transmit data to the > Host(s). So there is a single host port, but it can support multiple hosts, each having a subset of the available DMA channels. Maybe it is explain later, but why call it a _single_ host port? Apart from the DMA channels, are there other things the hosts are sharing? > The exchange of data occurs via TX/RX DMA Channels (Hardware > Queues). These Hardware Queues are a limited resource (8 TX Channels and > up to 64 RX Flows). If the Operating System on any of the cores is the > sole user of CPSW then all of these Hardware Queues can be claimed by that > OS. However, when CPSW has to be shared across the Operating Systems on > various cores with the aim of enabling Ethernet Functionality for the > Applications running on different cores, it is necessary to share these > Hardware Queues in a manner that prevents conflicts. On the control path > which corresponds to the configuration of CPSW to get it up and running, > since there is no Integrated Processor within CPSW that can be programmed > with a startup configuration, either the Operating System or Firmware > running on one of the cores has to take the responsibility of setting it. > One option in this case happens to be the Ethernet Switch Firmware (EthFw) > which is loaded by the Bootloader on a remote core at the same time that > Linux and other Operating Systems begin booting. EthFw quickly powers on > and configures CPSW getting the Forwarding Path functional. At some point, a definition of functional will be needed. How does the EthFw know what is required? Should Linux care? Can Linux change it? > Once Linux and > other Operating Systems on various cores are ready, they can communicate > with EthFw to obtain details of the Hardware Queues allocated to them to > exchange data with CPSW. > With the knowledge of the Hardware Queues that > have been allocated, Linux can use the DMA APIs to setup these queues > to exchange data with CPSW. This might be an important point. You communicate with the CPSW. You don't communicate transparently through the CPSW to external ports? There is no mechanism for a host to say, send this packet out port X? It is the CPSW which decides, based on its address tables? The destination MAC address decides where a packet goes. > Setting up the Hardware Queues alone isn't sufficient to exchange data > with the external network. Consider the following example: > The ethX interface in userspace which has been created to transmit/receive > data to/from CPSW has the user-assigned MAC Address of "M". The ping > command is run with the destination IP of "D". This results in an ARP > request sent from ethX which is transmitted out of all MAC Ports of CPSW > since it is a Broadcast request. Assuming that "D" is a valid > destination IP, the ARP reply is received on one of the MAC Ports which > is now a Unicast reply with the destination MAC Address of "M". The ALE > (Address Lookup Engine) in CPSW has learnt that the MAC Address "M" > corresponds to the Host Port when the ARP request was sent out. So the > Unicast reply isn't dropped. The challenge however is determining which > RX DMA Channel (Flow) to send the Unicast reply on. In the case of a > single Operating System owning all Hardware Queues, sending it on any of > the RX DMA Channels would have worked. In the current case where the RX > DMA Channels map to different Hosts (Operating Systems and Applications), > the mapping between the MAC Address "M" and the RX DMA Channel has to be > setup to ensure that the correct Host receives the ARP reply. This > necessitates a method to inform the MAC Address "M" associated with the > interface ethX to EthFw so that EthFw can setup the MAC Address "M" to > RX DMA Channel map accordingly. Why not have EthFW also do learning? The broadcast ARP request tells you that MAC address M is associated to a TX DMA channel. EthFW should know the Rx DMA channel which pairs with it, and can program ALE. That is how a switch works, it learns what MAC address is where, it is not told. > At this point, Linux can exchange data with the external network via CPSW, > but no device on the external network can initiate the communication by > itself unless it already has the ARP entry for the IP Address of ethX. > That's because CPSW doesn't support packet replication implying that any > Broadcast/Multicast packets received on the MAC Ports can only be sent > on one of the RX DMA Channels. That sounds broken. And this is where we need to be very careful. It is hard to build a generic model when the first device using it is broken. Ethernet switches have always been able to replicate. Dumb hubs did nothing but replicate. Address learning, and forwarding out specific ports came later, but multicast and broadcast was always replicated. IGMP snooping came later still, which reduced multicast replication. And your switch cannot do replication.... > So the Broadcast/Multicast packets can > only be received by one Host. Consider the following example: > A PC on the network tries to ping the IP Address of ethX. In both of the > following cases: > 1. Linux hasn't yet exchanged data with the PC via ethX. > 2. The MAC Address of ethX has changed. > the PC sends an ARP request to one of the MAC Ports on CPSW to figure > out the MAC Address of ethX. Since the ARP request is a Broadcast > request, it is not possible for CPSW to determine the correct Host, > since the Broadcast MAC isn't unique to any Host. So CPSW is forced > to send the Broadcast request to a preconfigured RX DMA Channel which > in this case happens to be the one mapped to EthFw. Thus, if EthFw > is aware of the IP Address of ethX, it can generate and send the ARP > reply containing the MAC Address "M" of ethX that it was informed of. > With this, the PC can initiate communication with Linux as well. > > Similarly, in the case of Multicast packets, if Linux wishes to receive > certain Multicast packets, it needs to inform the same to EthFw which > shall then replicate the Multicast packets it received from CPSW and > transmit them via alternate means (Shared Memory for example) to Linux. This all sounds like you are working around broken behaviour, not something generic. What i actually think you need to do is hide all the broken behaviour. Trap all multicast/broadcast to EthFw. It can run a software bridge, and do learning. It will see the outgoing ARP request from a host and learn the host MAC address. It can then flood the packet out the external ports, working around the CSPW brokeness. It can also program the ALE, so the reply goes straight to the host. Incoming broadcast and multicast is also trapped to the EthFW and it can use its software bridge to flood the packet to all the hosts. It can also perform IGMP snooping, and learn which hosts are interested in Multicast. Your switch then functions as a switch. And you are then the same as the RealTek and Samsung device. Linux is just a plain boring host connect to a switch, which somebody else is managing. No new model needed. > All data between Linux (Or any Operating System) and EthFw is exchanged > via the Hardware Mailboxes with the help of the RPMsg framework. Since > all the resource allocation information comes from EthFw, the > vendor-specific implementation in the Linux Client is limited to the DMA > APIs used to setup the Hardware Queues and to transmit/receive data with > the Ethernet Switch. Therefore, it might be possible to move most of the > vendor specific implementation to the Switch Configuration Firmware > (similar to EthFw), to make the Linux Client implementation as generic > and vendor agnostic as possible. I believe that this series more or less > does the same, just using custom terminology which can be made generic. This is actually very similar to what your college is doing: https://lore.kernel.org/netdev/20240531064006.1223417-1-y-mallik@ti.com/ The only real difference is shared memory vs DMA. Andrew
On Wed, Jun 12, 2024 at 12:03:00AM +0200, Andrew Lunn wrote: > > System Architecture and Implementation Details > > ============================================== > > > > The CPSW Ethernet Switch has a single Host Port (CPU facing port) through > > which it can receive data from the Host(s) and transmit data to the > > Host(s). > > So there is a single host port, but it can support multiple hosts, > each having a subset of the available DMA channels. Maybe it is > explain later, but why call it a _single_ host port? Apart from the > DMA channels, are there other things the hosts are sharing? The term __single__ is important here in the context of the questions you have asked below. Please consider the analogy of an external network switch to which our Laptop/PC is connected to via a LAN Cable. Such external network switches have multiple ports, all of which are treated identically. Devices connected to one port can communicate with devices connected on other ports of that network switch. So there technically isn't a "special port" or Host Port, since each Port connects to a different device. In the case of CPSW, a CPSW5G instance for example, has 4 MAC Ports (which are the equivalent of the ports present on the external network switch and accessible via the external world) and 1 Host Port. The single Host Port has multiple TX/RX DMA Channels to transmit/receive data. If these channels are shared across multiple cores, then yes, we do have multiple hosts exchanging data with CPSW via its single host port. All rules that apply to the MAC Ports or any Ethernet Switch Port for that matter, also apply to the CPSW's Host Port. One such rule which is significant in the current context happens to be that "duplicate" packets cannot be sent out of a single port. I shall refer to this again in my reply below. > > > The exchange of data occurs via TX/RX DMA Channels (Hardware > > Queues). These Hardware Queues are a limited resource (8 TX Channels and > > up to 64 RX Flows). If the Operating System on any of the cores is the > > sole user of CPSW then all of these Hardware Queues can be claimed by that > > OS. However, when CPSW has to be shared across the Operating Systems on > > various cores with the aim of enabling Ethernet Functionality for the > > Applications running on different cores, it is necessary to share these > > Hardware Queues in a manner that prevents conflicts. On the control path > > which corresponds to the configuration of CPSW to get it up and running, > > since there is no Integrated Processor within CPSW that can be programmed > > with a startup configuration, either the Operating System or Firmware > > running on one of the cores has to take the responsibility of setting it. > > One option in this case happens to be the Ethernet Switch Firmware (EthFw) > > which is loaded by the Bootloader on a remote core at the same time that > > Linux and other Operating Systems begin booting. EthFw quickly powers on > > and configures CPSW getting the Forwarding Path functional. > > At some point, a definition of functional will be needed. How does the > EthFw know what is required? Should Linux care? Can Linux change it? The term functional can be considered to be the equivalent of a "start-up" configuration that is present in the external network switches. The code that is flashed into the external network switch's non-volatile memory to setup and configure the switch on device power-on is responsible to get the switch functional. From a user's perspective, functional will imply that the devices connected to the ports on the external network switch are able to exchange data with one another (Forwarding Path). Therefore, Linux doesn't need to know about this and also cannot change the startup configuration performed by EthFw. > > > Once Linux and > > other Operating Systems on various cores are ready, they can communicate > > with EthFw to obtain details of the Hardware Queues allocated to them to > > exchange data with CPSW. > > > With the knowledge of the Hardware Queues that > > have been allocated, Linux can use the DMA APIs to setup these queues > > to exchange data with CPSW. > > This might be an important point. You communicate with the CPSW. You > don't communicate transparently through the CPSW to external ports? > There is no mechanism for a host to say, send this packet out port X? > It is the CPSW which decides, based on its address tables? The > destination MAC address decides where a packet goes. Yes, the host cannot/doesn't decide which MAC Port the packet goes out of. The ALE in CPSW is responsible for deciding that. This is identical to an external network switch. A PC which sends traffic to the port on the network switch cannot/doesn't tell the switch which port to send it out of. The network switch is supposed to learn and determine this. The DMA Channels provide a path to/from CPSW's Host Port for each Host. Please refer to the following illustration corresponding to the data movement from each of the Hosts to the CPSW's Host Port via the ALE and then out of the MAC Ports: ------- --------- --------- CONTROL-PATH |Linux| |AUTOSAR| | EthFW | ------------- ------- --------- --------- | | | | | | | | DATA TX RX TX RX TX RX | PATH => | | | | | | | (DMA) | | | | | | | | | | | | | | \ \ \ \ / / | \ \ \ \ / / | \ \ \ \ / / | \ \ \ \ / / | \ \ \ \ / / | =============================== | || CPSW HOST PORT || V =============================== ----------- | |CPSW | TX + RX |CONTROL | | |REGISTERS| | ----------- | =================== ||ALE & SWITCH CORE|| =================== / | | \ / | | \ / | | \ TX+RX | \ ------- / | \ \ / TX+RX TX+RX TX+RX / | \ \ ========== ========== ========== ========== |MAC Port 1| |MAC Port 2| |MAC Port 3| |MAC Port 4| ========== ========== ========== ========== > > > Setting up the Hardware Queues alone isn't sufficient to exchange data > > with the external network. Consider the following example: > > The ethX interface in userspace which has been created to transmit/receive > > data to/from CPSW has the user-assigned MAC Address of "M". The ping > > command is run with the destination IP of "D". This results in an ARP > > request sent from ethX which is transmitted out of all MAC Ports of CPSW > > since it is a Broadcast request. Assuming that "D" is a valid > > destination IP, the ARP reply is received on one of the MAC Ports which > > is now a Unicast reply with the destination MAC Address of "M". The ALE > > (Address Lookup Engine) in CPSW has learnt that the MAC Address "M" > > corresponds to the Host Port when the ARP request was sent out. So the > > Unicast reply isn't dropped. The challenge however is determining which > > RX DMA Channel (Flow) to send the Unicast reply on. In the case of a > > single Operating System owning all Hardware Queues, sending it on any of > > the RX DMA Channels would have worked. In the current case where the RX > > DMA Channels map to different Hosts (Operating Systems and Applications), > > the mapping between the MAC Address "M" and the RX DMA Channel has to be > > setup to ensure that the correct Host receives the ARP reply. This > > necessitates a method to inform the MAC Address "M" associated with the > > interface ethX to EthFw so that EthFw can setup the MAC Address "M" to > > RX DMA Channel map accordingly. > > Why not have EthFW also do learning? The broadcast ARP request tells > you that MAC address M is associated to a TX DMA channel. EthFW should > know the Rx DMA channel which pairs with it, and can program ALE. > > That is how a switch works, it learns what MAC address is where, it is > not told. The ALE in CPSW learns just like any other switch and doesn't need to be programmed. When the ARP broadcast is sent out via the ALE, it learns that the MAC Address "M" is from the Host Port. The problem however is that knowing this alone isn't sufficient for the return path (ARP reply). The ARP reply contains the destination MAC Address "M" which tells the ALE that the Host Port is the destination. So the ALE will send the ARP reply to the Host Port. But, as the illustration shows, from the Host Port, there are multiple RX DMA Channels for each Host. So the ALE did its job as expected from any standard ALE. The missing part here is determining which RX DMA Channel (Flow) to place that packet on. That requires programming the Classifiers in CPSW to map that packets containing the MAC Address "M" to the RX DMA Flow corresponding to the Host which has registered with that MAC Address "M". This is handled by EthFw. EthFw doesn't/cannot snoop on all traffic on the Host Port, since it doesn't lie in between the Host Port and the other Hosts. Rather, it is quite similar to a Host itself since it also has dedicated TX/RX DMA Channels to exchange traffic with CPSW. > > > At this point, Linux can exchange data with the external network via CPSW, > > but no device on the external network can initiate the communication by > > itself unless it already has the ARP entry for the IP Address of ethX. > > That's because CPSW doesn't support packet replication implying that any > > Broadcast/Multicast packets received on the MAC Ports can only be sent > > on one of the RX DMA Channels. > > That sounds broken. > > And this is where we need to be very careful. It is hard to build a > generic model when the first device using it is broken. Ethernet > switches have always been able to replicate. Dumb hubs did nothing but > replicate. Address learning, and forwarding out specific ports came > later, but multicast and broadcast was always replicated. IGMP > snooping came later still, which reduced multicast replication. > > And your switch cannot do replication.... I will respectfully disagree with your statement here. Packet replication is supported in CPSW just like any other Ethernet Switch which can create copies of Broadcast/Multicast traffic it receives on one "Port" and send them out of the other "Ports". That is exactly how the ARP Broadcast request sent from Linux to the CPSW Host Port is duplicated in the Switch Core and is sent out via all of the MAC Ports in the illustration above. The "packet replication" that I was referring to, is that of creating duplicates of a packet and sending them out on the same "Port" (Host Port in this case). This is not expected of any Ethernet Switch and can only be considered an optional feature. In the current case, due to the presence of Multiple Hosts connected to a __single__ Host Port, copies of Broadcast or Multicast traffic are expected on a single Port so that one copy each of the Broadcast/Multicast traffic directed to the Host Port can be sent on each of the RX DMA Flows for every Host. Since that is not supported, all Broadcast/Multicast traffic directed to the Host Port from the Switch Core is by default placed on the RX DMA Flow corresponding to EthFw. EthFw then creates copies of these in software and shares them with each Host via Shared Memory for example. > > > So the Broadcast/Multicast packets can > > only be received by one Host. Consider the following example: > > A PC on the network tries to ping the IP Address of ethX. In both of the > > following cases: > > 1. Linux hasn't yet exchanged data with the PC via ethX. > > 2. The MAC Address of ethX has changed. > > the PC sends an ARP request to one of the MAC Ports on CPSW to figure > > out the MAC Address of ethX. Since the ARP request is a Broadcast > > request, it is not possible for CPSW to determine the correct Host, > > since the Broadcast MAC isn't unique to any Host. So CPSW is forced > > to send the Broadcast request to a preconfigured RX DMA Channel which > > in this case happens to be the one mapped to EthFw. Thus, if EthFw > > is aware of the IP Address of ethX, it can generate and send the ARP > > reply containing the MAC Address "M" of ethX that it was informed of. > > With this, the PC can initiate communication with Linux as well. > > > > Similarly, in the case of Multicast packets, if Linux wishes to receive > > certain Multicast packets, it needs to inform the same to EthFw which > > shall then replicate the Multicast packets it received from CPSW and > > transmit them via alternate means (Shared Memory for example) to Linux. > > This all sounds like you are working around broken behaviour, not > something generic. > > What i actually think you need to do is hide all the broken > behaviour. Trap all multicast/broadcast to EthFw. It can run a All Multicast/Broadcast traffic received by the Host Port is trapped and sent to EthFw like I have mentioned in my reply above. > software bridge, and do learning. It will see the outgoing ARP request > from a host and learn the host MAC address. It can then flood the As I have mentioned earlier, EthFw is not in the path between the CPSW Host Port and the Hosts. So your suggestion is not applicable. > packet out the external ports, working around the CSPW brokeness. It > can also program the ALE, so the reply goes straight to the > host. Incoming broadcast and multicast is also trapped to the EthFW > and it can use its software bridge to flood the packet to all the > hosts. It can also perform IGMP snooping, and learn which hosts are > interested in Multicast. > > Your switch then functions as a switch. > > And you are then the same as the RealTek and Samsung device. Linux is > just a plain boring host connect to a switch, which somebody else is > managing. No new model needed. I hope that the illustration above along with my replies would have clarified that CPSW is *not* broken and works just like any Ethernet Switch is expected to work. It is only because there is a __single__ Host Port from CPSW that is shared across Hosts, that the EthFw based model is required. If there were dedicated Host Ports for each Host, then just like the Broadcast/Multicast traffic is already replicated for each Port (whether it is a MAC Port or the Host Port), the traffic would also be replicated in CPSW for each Host Port. > > > All data between Linux (Or any Operating System) and EthFw is exchanged > > via the Hardware Mailboxes with the help of the RPMsg framework. Since > > all the resource allocation information comes from EthFw, the > > vendor-specific implementation in the Linux Client is limited to the DMA > > APIs used to setup the Hardware Queues and to transmit/receive data with > > the Ethernet Switch. Therefore, it might be possible to move most of the > > vendor specific implementation to the Switch Configuration Firmware > > (similar to EthFw), to make the Linux Client implementation as generic > > and vendor agnostic as possible. I believe that this series more or less > > does the same, just using custom terminology which can be made generic. > > This is actually very similar to what your college is doing: > > https://lore.kernel.org/netdev/20240531064006.1223417-1-y-mallik@ti.com/ > > The only real difference is shared memory vs DMA. Yes, the Shared Memory path is intended for the low-bandwidth Broadcast/Multicast traffic from EthFw while the DMA path is dedicated for high-bandwidth Unicast traffic. The current series implements the DMA path while the other series you have referred to implements the Shared Memory path. Both of them together enable the desired functionality. Regards, Siddharth.
> The DMA Channels provide a path to/from CPSW's Host Port for each Host. > > Please refer to the following illustration corresponding to the data > movement from each of the Hosts to the CPSW's Host Port via the ALE and > then out of the MAC Ports: > > ------- --------- --------- CONTROL-PATH > |Linux| |AUTOSAR| | EthFW | ------------- > ------- --------- --------- | > | | | | | | | > DATA TX RX TX RX TX RX | > PATH => | | | | | | | > (DMA) | | | | | | | > | | | | | | | > \ \ \ \ / / | > \ \ \ \ / / | > \ \ \ \ / / | > \ \ \ \ / / | > \ \ \ \ / / | > =============================== | > || CPSW HOST PORT || V > =============================== ----------- > | |CPSW | > TX + RX |CONTROL | > | |REGISTERS| > | ----------- > | > =================== > ||ALE & SWITCH CORE|| > =================== > / | | \ > / | | \ > / | | \ > TX+RX | \ ------- > / | \ \ > / TX+RX TX+RX TX+RX > / | \ \ > ========== ========== ========== ========== > |MAC Port 1| |MAC Port 2| |MAC Port 3| |MAC Port 4| > ========== ========== ========== ========== So, in summary, you have one host port, and on top of that a number of virtual ports. Because of limitations in the ALE, those virtual ports don't work in the same way as real ports, replication is not possible, nor learning for individual virtual ports. The typical 1990 solution to that would be to flood packets to all hosts, and let them filter out the packets they are not interested in. 1990 Ethernet was a shared medium, you expect to see packets for other hosts. But the hardware also cannot do that. So you have to program the classify to augment the ALE, and the classifier is the one that decides which virtual port a packet goes out. But the classifier does not perform learning. You need additional mechanisms to program that classifier. > Host which has registered with that MAC Address "M". This is handled by > EthFw. EthFw doesn't/cannot snoop on all traffic on the Host Port, since > it doesn't lie in between the Host Port and the other Hosts. Rather, it > is quite similar to a Host itself since it also has dedicated TX/RX DMA > Channels to exchange traffic with CPSW. I did not say snoop. I said trap. There is a difference. Snoop would be it sees the packet, as it going by. Trap means it actually gets passed the packet, and it needs to deal with it, decide the outgoing port. So i would trap all broadcast and multicast from the virtual ports to the EthFW. Let the EthFw deal with that traffic, perform learning, and programming the classifier, and flood it out user ports for broadcast, or unicast out specific ports for multicast where IGMP snooping indicates it should go. > Since that is not supported, all > Broadcast/Multicast traffic directed to the Host Port from the Switch Core > is by default placed on the RX DMA Flow corresponding to EthFw. EthFw then > creates copies of these in software and shares them with each Host via > Shared Memory for example. Why shared memory? EthFw needs to be able to direct packets out specific virtual ports otherwise it cannot do {R}STP, PTP, IGMP snooping etc. So it should just pass the packet back to the CPSW, which will hairpin the packet, hit the classifier, and then send it out the correct virtual port to the correct host. > Yes, the Shared Memory path is intended for the low-bandwidth > Broadcast/Multicast traffic from EthFw while the DMA path is dedicated > for high-bandwidth Unicast traffic. The current series implements the > DMA path while the other series you have referred to implements the > Shared Memory path. Both of them together enable the desired functionality. So i think we are agreed a new model is not needed. Linux is just a host connected to a managed switch. Linux has no roll in managing that switch, and has no idea about the ports of that switch. It is just an end system, running end system software. You need a 'MAC' driver in Linux, so Linux sees just a normal network interface. And it must see a single MAC driver, so if you really do need to use shared memory in parallel to DMA, you will need to combine that into one driver. Andrew
diff --git a/Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst b/Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst new file mode 100644 index 000000000000..46bff67b3446 --- /dev/null +++ b/Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst @@ -0,0 +1,182 @@ +.. SPDX-License-Identifier: GPL-2.0-only or MIT + +========================================== +Texas Instruments CPSW Proxy Client driver +========================================== + +Introduction +============ + +The CPSW (Common Platform Switch) Ethernet Switch on TI's K3 SoCs provides +Ethernet functionality. There may be multiple instances of CPSW on a single +SoC. The term "CPSWnG" is used to indicate the number of MAC Ports supported +by a specific instance of CPSW. CPSWnG indicates that the peripheral has +(n-1) MAC Ports and 1 Host Port. Examples of existing instances are: +CPSW2G => 1 MAC Port and 1 Host Port +CPSW3G => 2 MAC Ports and 1 Host Port +CPSW5G => 4 MAC Ports and 1 Host Port +CPSW9G => 8 MAC Ports and 1 Host Port + +The presence of 2 or more MAC Ports implies that Hardware Switching can +be enabled between the MAC Ports if required. + +The "am65-cpsw-nuss.c" driver in Linux at: +drivers/net/ethernet/ti/am65-cpsw-nuss.c +provides Ethernet functionality for applications on Linux. +It also handles both the control-path and data-path, namely: +Control => Configuration of the CPSW Peripheral +Data => Configuration of the DMA Channels to transmit/receive data + +The aforementioned configuration supports use-cases where all applications +which require Ethernet functionality are only running on Linux. + +However, there are use-cases where applications running on different +Operating Systems across multiple cores on the SoC require Ethernet +functionality. Such use-cases can be supported by implementing a +Client-Server model to share the data-path among Clients while the Server +owns the control-path. + +On TI's K3 SoCs (J721E, J7200 and J784S4 in particular), the Ethernet Switch +Firmware (EthFw) running on the MAIN R5F core acts as the Server and +configures the CPSWnG instance (CPSW5G on J7200 and CPSW9G on J721E, J784S4) +of the CPSW Ethernet Switch on the SoC. The Clients running on various cores +communicate with EthFw via RPMsg (Remote Processor Messaging) to request +resource allocation details during initialization, followed by requesting +EthFw to enable features supported by CPSW based on the features required +by the applications running on the respective cores. + +EthFw handles requests from the Clients and evaluates them before configuring +CPSW based on the request. Since no Client is actually in control of CPSW and +only requests EthFw for configuring CPSW, EthFw acts as the proxy for the +Clients. Thus, the Linux Client which interfaces with EthFw is named: +CPSW Proxy Client + +The data-path for the CPSW Proxy Client driver remains identical to the +"am65-cpsw-nuss.c" driver which happens to be DMA. It is only the control-path +that is different. + +Client-Server discovery occurs over the RPMsg-Bus. EthFw announces its +RPMsg Endpoint name over the RPMsg-Bus. The CPSW Proxy Client driver +registers itself with the Linux RPMsg framework to be probed for the same +Endpoint name. Following probe, the Linux Client driver begins communicating +with EthFw and queries details of the resources available for the Linux Client. + +Terminology +=========== + +Virtual Port + A Virtual Port refers to the Software View of an Ethernet MAC Port. + There are two types of Virtual Ports: + 1. Virtual MAC Only Port + 2. Virtual Switch Port + +Virtual MAC Only Port + A Virtual MAC only Port refers to a dedicated physical MAC Port for + a Client. This corresponds to MAC Mode of operation in Ethernet + Terminology. All traffic sent to or received from the Physical + MAC Port is that of the Client to which the Virtual MAC Only Port + has been allocated. + +Virtual Switch Port + A Virtual Switch Port refers to a group of physical MAC ports with + Switching enabled across them. This implies that any traffic sent + to the Port from a Client could potentially exit a Physical MAC + Port along with the traffic from other Clients. Similarly, the traffic + received on the Port by a Client could have potentially ingressed + on a Physical MAC Port along with the traffic meant for other Clients. + While the ALE (Address Lookup Engine) handles segregating the traffic, + and the CPSW Ethernet Switch places traffic on dedicated RX DMA Flows + meant for a single Client, it is worth noting that the bandwidths + of the Physical MAC Port are shared by Clients when traffic is sent to + or received from a Virtual Switch Port. + +Network Interface + The user-visible interface in Linux userspace exposed to applications + that serves as the entry/exit point for traffic to/from the Virtual + Ports. A single network interface (ethX) maps to either a Virtual + MAC Only Port or a Virtual Switch Port. + +C2S + RPMsg source is Client and destination is Server. + +S2C + RPMsg source is Server and destination is Client. + +Initialization Sequence +======================= + +The sequence of message exchanges between the Client driver and EthFw starting +from the driver probe and ending with the interfaces being brought up is as +follows: +1. C2S ETHFW_VIRT_PORT_INFO requesting details of Virtual Ports available + for the Linux Client. +2. S2C response containing requested details +3. C2S ETHFW_VIRT_PORT_ATTACH request for each Virtual Port allocated during + step 2. +4. S2C response containing details of the MTU Size, number of Tx DMA Channels + and RX DMA Flows for the specified Virtual Port. The *Features* associated + with the Virtual Port are also shared such as Multicast Filtering capability. +5. C2S ETHFW_ALLOC_RX request for each RX DMA Flow for a Virtual Port. +6. S2C response containing details of the RX PSI-L Thread ID, Flow base and + Flow offset. +7. C2S ETHFW_ALLOC_TX request for each TX DMA Channel for a Virtual Port. +8. S2C response containing details of the TX PSI-L Thread ID. +9. C2S ETHFW_ALLOC_MAC request for each Virtual Port. +10. S2C response containing the MAC Address corresponding to the Virtual Port. +11. C2S ETHFW_MAC_REGISTER request for each Virtual Port with the MAC Address + allocated in step 10. This is necessary to steer packets that ingress on + the MAC Ports of CPSW onto the RX DMA Flow for the Virtual Port in order + to allow the Client to receive the packets. +12. S2C response indicating status of request. +13. C2S ETHFW_IPv4_REGISTER request *only* for Virtual Switch Port interface. + The IPv4 address assigned to the "ethX" network interface in Linux + corresponding to the Virtual Switch Port interface has to be registered + with EthFw. This is due to the reason that all Broadcast requests including + ARP requests received by the MAC Ports corresponding to the Virtual Switch + Port are consumed solely be EthFw. Such traffic is sent to Clients by + alternate methods. Therefore EthFw needs to know the IPv4 address for the + "ethX" network interface in Linux in order to automatically respond to + ARP requests, thereby enabling Unicast communication. +14. S2C response indicating status of request. +15. C2S ETHFW_MCAST_FILTER_ADD request to register the Multicast Addresses + associated with the network interface corresponding to the Virtual Port + which has the Multicast Filtering capability. +16. S2C response indicating status of request. +17. C2S ETHFW_MCAST_FILTER_DEL request to deregister the Multicast Addresses + associated with the network interface corresponding to the Virtual Port + which has the Multicast Filtering capability. +18. S2C response indicating status of request. + +Shutdown Sequence +================= + +The sequence of message exchanges between the Client driver and EthFw on module +removal are as follows: +1. C2S ETHFW_MAC_DEREGISTER request to deregister the MAC Address for each + Virtual Port. +2. S2C response indicating status of request. +3. C2S ETHFW_MCAST_FILTER_DEL request to deregister the Multicast Addresses + associated with the network interface corresponding to the Virtual Port + which has the Multicast Filtering capability. +4. S2C response indicating status of request. +5. C2S ETHFW_FREE_MAC request to release the MAC Address allocated to each + Virtual Port. +6. S2C response indicating status of request. +7. C2S ETHFW_FREE_TX request to release the TX DMA Channel for each TX Channel + for every Virtual Port. +8. S2C response indicating status of request. +9. C2S ETHFW_FREE_RX request to release the RX DMA Flow for each RX Channel + for every Virtual Port. +10. S2C response indicating status of request. +11. C2S ETHFW_VIRT_PORT_DETACH request to release each Virtual Port. +12. S2C response indicating status of request. + +Features Supported +================== + +The set of features supported in addition to providing basic Ethernet +Functionality are: +1. Multicast Filtering +2. Determining Link Status of the network interface corresponding to the + Virtual MAC Only port via ethtool. +3. Interrupt Pacing/Coalescing
The CPSW Proxy Client driver interfaces with Ethernet Switch Firmware on a remote core to enable Ethernet functionality for applications running on Linux. The Ethernet Switch Firmware (EthFw) is in control of the CPSW Ethernet Switch on the SoC and acts as the Server, offering services to Clients running on various cores. Signed-off-by: Siddharth Vadapalli <s-vadapalli@ti.com> --- .../ethernet/ti/cpsw_proxy_client.rst | 182 ++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 Documentation/networking/device_drivers/ethernet/ti/cpsw_proxy_client.rst