| Message ID | 20230710223304.1174642-1-almasrymina@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Device Memory TCP | expand |
On 7/10/23 15:32, Mina Almasry wrote: > * TL;DR: > > Device memory TCP (devmem TCP) is a proposal for transferring data to and/or > from device memory efficiently, without bouncing the data to a host memory > buffer. (I'm writing this as someone who might plausibly use this mechanism, but I don't think I'm very likely to end up working on the kernel side, unless I somehow feel extremely inspired to implement it for i40e.) I looked at these patches and the GVE tree, and I'm trying to wrap my head around the data path. As I understand it, for RX: 1. The GVE driver notices that the queue is programmed to use devmem, and it programs the NIC to copy packet payloads to the devmem that has been programmed. 2. The NIC receives the packet and copies the header to kernel memory and the payload to dma-buf memory. 3. The kernel tells userspace where in the dma-buf the data is. 4. Userspace does something with the data. 5. Userspace does DONTNEED to recycle the memory and make it available for new received packets. Did I get this right? This seems a bit awkward if there's any chance that packets not intended for the target device end up in the rxq. I'm wondering if a more capable if somewhat higher latency model could work where the NIC stores received packets in its own device memory. Then userspace (or the kernel or a driver or whatever) could initiate a separate DMA from the NIC to the final target *after* reading the headers. Can the hardware support this? Another way of putting this is: steering received data to a specific device based on the *receive queue* forces the logic selecting a destination device to be the same as the logic selecting the queue. RX steering logic is pretty limited on most hardware (as far as I know -- certainly I've never had much luck doing anything especially intelligent with RX flow steering, and I've tried on a couple of different brands of supposedly fancy NICs). But Linux has very nice capabilities to direct packets, in software, to where they are supposed to go, and it would be nice if all that logic could just work, scalably, with device memory. If Linux could examine headers *before* the payload gets DMAed to wherever it goes, I think this could plausibly work quite nicely. One could even have an easy-to-use interface in which one directs a *socket* to a PCIe device. I expect, although I've never looked at the datasheets, that the kernel could even efficiently make rx decisions based on data in device memory on upcoming CXL NICs where device memory could participate in the host cache hierarchy. My real ulterior motive is that I think it would be great to use an ability like this for DPDK-like uses. Wouldn't it be nifty if I could open a normal TCP socket, then, after it's open, ask the kernel to kindly DMA the results directly to my application memory (via udmabuf, perhaps)? Or have a whole VLAN or macvlan get directed to a userspace queue, etc? It also seems a bit odd to me that the binding from rxq to dma-buf is established by programming the dma-buf. This makes the security model (and the mental model) awkward -- this binding is a setting on the *queue*, not the dma-buf, and in a containerized or privilege-separated system, a process could have enough privilege to make a dma-buf somewhere but not have any privileges on the NIC. (And may not even have the NIC present in its network namespace!) --Andy
On Sun, 16 Jul 2023 19:41:28 -0700 Andy Lutomirski wrote: > I'm wondering if a more capable if somewhat higher latency model could > work where the NIC stores received packets in its own device memory. > Then userspace (or the kernel or a driver or whatever) could initiate a > separate DMA from the NIC to the final target *after* reading the > headers. Can the hardware support this? No, no, that's impossible. SW response times are in 100s of usec (at best) which at 200Gbps already means megabytes of data _per-queue_. Way more than the amount of buffer NICs will have. The Rx application can bind to a IP addr + Port and then install a one-sided-3-tuple (dst IP+proto+port) rule in the HW. Worst case a full 5-tuple per flow. Most NICs support OvS offloads with 100s of thousands of flows. The steering should be bread and butter. It does require splitting flows into separate data and control channels, but it's the right trade-off - complexity should be on the SW side.
On Sun, Jul 16, 2023 at 7:41 PM Andy Lutomirski <luto@kernel.org> wrote: > > On 7/10/23 15:32, Mina Almasry wrote: > > * TL;DR: > > > > Device memory TCP (devmem TCP) is a proposal for transferring data to and/or > > from device memory efficiently, without bouncing the data to a host memory > > buffer. > > (I'm writing this as someone who might plausibly use this mechanism, but > I don't think I'm very likely to end up working on the kernel side, > unless I somehow feel extremely inspired to implement it for i40e.) > > I looked at these patches and the GVE tree, and I'm trying to wrap my > head around the data path. As I understand it, for RX: > > 1. The GVE driver notices that the queue is programmed to use devmem, > and it programs the NIC to copy packet payloads to the devmem that has > been programmed. > 2. The NIC receives the packet and copies the header to kernel memory > and the payload to dma-buf memory. > 3. The kernel tells userspace where in the dma-buf the data is. > 4. Userspace does something with the data. > 5. Userspace does DONTNEED to recycle the memory and make it available > for new received packets. > > Did I get this right? > Sorry for the late reply. I'm a bit buried working on the follow up to this proposal: exploring using dma-bufs without pages. Yes, this is completely correct. > This seems a bit awkward if there's any chance that packets not intended > for the target device end up in the rxq. > It does a bit. What happens in practice is that we use RSS to steer general traffic away from the devmem queues, and we use flow steering to steer specific flows to devem queues. In the case where the RSS/flow steering configuration is done incorrectly, the user would call recvmsg() on a devmem skb and if they haven't specified the MSG_SOCK_DEVMEM flag they'd get an error. > I'm wondering if a more capable if somewhat higher latency model could > work where the NIC stores received packets in its own device memory. > Then userspace (or the kernel or a driver or whatever) could initiate a > separate DMA from the NIC to the final target *after* reading the > headers. Can the hardware support this? > Not that I know of. I guess Jakub also responded with the same. > Another way of putting this is: steering received data to a specific > device based on the *receive queue* forces the logic selecting a > destination device to be the same as the logic selecting the queue. RX > steering logic is pretty limited on most hardware (as far as I know -- > certainly I've never had much luck doing anything especially intelligent > with RX flow steering, and I've tried on a couple of different brands of > supposedly fancy NICs). But Linux has very nice capabilities to direct > packets, in software, to where they are supposed to go, and it would be > nice if all that logic could just work, scalably, with device memory. > If Linux could examine headers *before* the payload gets DMAed to > wherever it goes, I think this could plausibly work quite nicely. One > could even have an easy-to-use interface in which one directs a *socket* > to a PCIe device. I expect, although I've never looked at the > datasheets, that the kernel could even efficiently make rx decisions > based on data in device memory on upcoming CXL NICs where device memory > could participate in the host cache hierarchy. > > My real ulterior motive is that I think it would be great to use an > ability like this for DPDK-like uses. Wouldn't it be nifty if I could > open a normal TCP socket, then, after it's open, ask the kernel to > kindly DMA the results directly to my application memory (via udmabuf, > perhaps)? Or have a whole VLAN or macvlan get directed to a userspace > queue, etc? > > > It also seems a bit odd to me that the binding from rxq to dma-buf is > established by programming the dma-buf. That is specific to this proposal, and will likely be very different in future ones. I thought the dma-buf pages approach was extensible and the uapi belonged somewhere in dma-buf. Clearly not. The next proposal, I think, will program the rxq via some net uapi and will take the dma-buf as input. Probably some netlink api (not sure if ethtool family or otherwise). I'm working out details of this non-paged networking first. > This makes the security model > (and the mental model) awkward -- this binding is a setting on the > *queue*, not the dma-buf, and in a containerized or privilege-separated > system, a process could have enough privilege to make a dma-buf > somewhere but not have any privileges on the NIC. (And may not even > have the NIC present in its network namespace!) > > --Andy -- Thanks, Mina
On Tue, Jul 18, 2023 at 10:36:52AM -0700, Mina Almasry wrote: > That is specific to this proposal, and will likely be very different > in future ones. I thought the dma-buf pages approach was extensible > and the uapi belonged somewhere in dma-buf. Clearly not. The next > proposal, I think, will program the rxq via some net uapi and will > take the dma-buf as input. Probably some netlink api (not sure if > ethtool family or otherwise). I'm working out details of this > non-paged networking first. In practice you want the application to startup, get itself some 3/5 tuples and then request the kernel to setup the flow steering and provision the NIC queues. This is the right moment for the application to provide the backing for the rx queue memory via a DMABUF handle. Ideally this would all be accessible to non-priv applications as well, so I think you'd want some kind of system call that sets all this up and takes in a FD for the 3/5-tuple socket (to prove ownership over the steering) and the DMABUF FD. The queues and steering should exist only as long as the application is still running (whatever that means). Otherwise you have a big mess to clean up whenever anything crashes. netlink feels like a weird API choice for that, in particular it would be really wrong to somehow bind the lifecycle of a netlink object to a process. Further, if you are going to all the trouble of doing this, it seems to me you should make it work with any kind of memory, including CPU memory. Get a consistent approach to zero-copy TCP RX. So also allow a memfd or similar to be passed in as the backing storage. Jason
On Tue, 18 Jul 2023 15:06:29 -0300 Jason Gunthorpe wrote: > netlink feels like a weird API choice for that, in particular it would > be really wrong to somehow bind the lifecycle of a netlink object to a > process. Netlink is the right API, life cycle of objects can be easily tied to a netlink socket.
On 7/18/23 12:15 PM, Jakub Kicinski wrote: > On Tue, 18 Jul 2023 15:06:29 -0300 Jason Gunthorpe wrote: >> netlink feels like a weird API choice for that, in particular it would >> be really wrong to somehow bind the lifecycle of a netlink object to a >> process. > > Netlink is the right API, life cycle of objects can be easily tied to > a netlink socket. That is an untuitive connection -- memory references, h/w queues, flow steering should be tied to the datapath socket, not a control plane socket.
On Tue, 18 Jul 2023 12:20:59 -0600 David Ahern wrote: > On 7/18/23 12:15 PM, Jakub Kicinski wrote: > > On Tue, 18 Jul 2023 15:06:29 -0300 Jason Gunthorpe wrote: > >> netlink feels like a weird API choice for that, in particular it would > >> be really wrong to somehow bind the lifecycle of a netlink object to a > >> process. > > > > Netlink is the right API, life cycle of objects can be easily tied to > > a netlink socket. > > That is an untuitive connection -- memory references, h/w queues, flow > steering should be tied to the datapath socket, not a control plane socket. There's one RSS context for may datapath sockets. Plus a lot of the APIs already exist, and it's more of a question of packaging them up at the user space level. For things which do not have an API, however, netlink, please.
On 7/18/23 12:29 PM, Jakub Kicinski wrote: > On Tue, 18 Jul 2023 12:20:59 -0600 David Ahern wrote: >> On 7/18/23 12:15 PM, Jakub Kicinski wrote: >>> On Tue, 18 Jul 2023 15:06:29 -0300 Jason Gunthorpe wrote: >>>> netlink feels like a weird API choice for that, in particular it would >>>> be really wrong to somehow bind the lifecycle of a netlink object to a >>>> process. >>> >>> Netlink is the right API, life cycle of objects can be easily tied to >>> a netlink socket. >> >> That is an untuitive connection -- memory references, h/w queues, flow >> steering should be tied to the datapath socket, not a control plane socket. > > There's one RSS context for may datapath sockets. Plus a lot of the > APIs already exist, and it's more of a question of packaging them up > at the user space level. For things which do not have an API, however, > netlink, please. I do not see how 1 RSS context (or more specifically a h/w Rx queue) can be used properly with memory from different processes (or dma-buf references). When the process dies, that memory needs to be flushed from the H/W queues. Queues with interlaced submissions make that more complicated. I guess the devil is in the details; I look forward to the evolution of the patches.
On Tue, 18 Jul 2023 16:35:17 -0600 David Ahern wrote: > I do not see how 1 RSS context (or more specifically a h/w Rx queue) can > be used properly with memory from different processes (or dma-buf > references). When the process dies, that memory needs to be flushed from > the H/W queues. Queues with interlaced submissions make that more > complicated. Agreed, one process, one control path socket. FWIW the rtnetlink use of netlink is very basic. genetlink already has some infra which allows associate state with a user socket and cleaning it up when the socket gets closed. This needs some improvements. A bit of a chicken and egg problem, I can't make the improvements until there are families making use of it, and nobody will make use of it until it's in tree... But the basics are already in place and I can help with building it out. > I guess the devil is in the details; I look forward to the evolution of > the patches. +1
On Tue, Jul 18, 2023 at 3:45 PM Jakub Kicinski <kuba@kernel.org> wrote: > > On Tue, 18 Jul 2023 16:35:17 -0600 David Ahern wrote: > > I do not see how 1 RSS context (or more specifically a h/w Rx queue) can > > be used properly with memory from different processes (or dma-buf > > references). Right, my experience with dma-bufs from GPUs are that they're allocated from the userspace and owned by the process that allocated the backing GPU memory and generated the dma-buf from it. I.e., we're limited to 1 dma-buf per RX queue. If we enable binding multiple dma-bufs to the same RX queue, we have a problem, because AFAIU the NIC can't decide which dma-buf to put the packet into (it hasn't parsed the packet's destination yet). > > When the process dies, that memory needs to be flushed from > > the H/W queues. Queues with interlaced submissions make that more > > complicated. > When the process dies, do we really want to flush the memory from the hardware queues? The drivers I looked at don't seem to have a function to flush the rx queues alone, they usually do an entire driver reset to achieve that. Not sure if that's just convenience or there is some technical limitation there. Do we really want to trigger a driver reset at the event a userspace process crashes? > Agreed, one process, one control path socket. > > FWIW the rtnetlink use of netlink is very basic. genetlink already has > some infra which allows associate state with a user socket and cleaning > it up when the socket gets closed. This needs some improvements. A bit > of a chicken and egg problem, I can't make the improvements until there > are families making use of it, and nobody will make use of it until > it's in tree... But the basics are already in place and I can help with > building it out. > I had this approach in mind (which doesn't need netlink improvements) for the next POC. It's mostly inspired by the comments from the cover letter of Jakub's memory-provider RFC, if I understood it correctly. I'm sure there's going to be some iteration, but roughly: 1. A netlink CAP_NET_ADMIN API which binds the dma-buf to any number of rx queues on 1 NIC. It will do the dma_buf_attach() and dma_buf_map_attachment() and leave some indicator in the struct net_device to tell the NIC that it's bound to a dma-buf. The actual binding doesn't actuate until the next driver reset. The API, I guess, can cause a driver reset (or just a refill of the rx queues, if you think that's feasible) as well to streamline things a bit. The API returns a file handle to the user representing that binding. 2. On the driver reset, the driver notices that its struct net_device is bound to a dma-buf, and sets up the dma-buf memory-provider instead of the basic one which provides host memory. 3. The user can close the file handle received in #1 to unbind the dma-buf from the rx queues. Or if the user crashes, the kernel closes the handle for us. The unbind doesn't take effect until the next flushing or rx queues, or the next driver reset (not sure the former is feasible). 4. The dma-buf memory provider keeps the dma buf mapping alive until the next driver reset, where all the dma-buf slices are freed, and the dma buf attachment mapping can be unmapped. I'm thinking the user sets up RSS and flow steering outside this API using existing ethtool APIs, but things can be streamlined a bit by doing some of these RSS/flow steering steps in cohesion with the dma-buf binding/unbinding. The complication with setting up flow steering in cohesion with dma-buf bind unbind is that the application may start more connections after the bind, and it will need to install flow steering rules for those too, and use the ethtool api for that. May as well use the ethtool apis for all of it...? From Jakub and David's comments it sounds (if I understood correctly), you'd like to tie the dma-buf bind/unbind functions to the lifetime of a netlink socket, rather than a struct file like I was thinking. That does sound cleaner, but I'm not sure how. Can you link me to any existing code examples? Or rough pointers to any existing code? > > I guess the devil is in the details; I look forward to the evolution of > > the patches. > > +1
On Wed, 19 Jul 2023 08:10:58 -0700 Mina Almasry <almasrymina@google.com> wrote: > On Tue, Jul 18, 2023 at 3:45 PM Jakub Kicinski <kuba@kernel.org> wrote: > > > > On Tue, 18 Jul 2023 16:35:17 -0600 David Ahern wrote: > > > I do not see how 1 RSS context (or more specifically a h/w Rx queue) can > > > be used properly with memory from different processes (or dma-buf > > > references). > > Right, my experience with dma-bufs from GPUs are that they're > allocated from the userspace and owned by the process that allocated > the backing GPU memory and generated the dma-buf from it. I.e., we're > limited to 1 dma-buf per RX queue. If we enable binding multiple > dma-bufs to the same RX queue, we have a problem, because AFAIU the > NIC can't decide which dma-buf to put the packet into (it hasn't > parsed the packet's destination yet). > > > > When the process dies, that memory needs to be flushed from > > > the H/W queues. Queues with interlaced submissions make that more > > > complicated. > > > > When the process dies, do we really want to flush the memory from the > hardware queues? The drivers I looked at don't seem to have a function > to flush the rx queues alone, they usually do an entire driver reset > to achieve that. Not sure if that's just convenience or there is some > technical limitation there. Do we really want to trigger a driver > reset at the event a userspace process crashes? Naive idea. Would it be possible for process to use mmap() on the GPU memory and then do zero copy TCP receive some how? Or is this what is being proposed.
On Wed, 19 Jul 2023 08:10:58 -0700 Mina Almasry wrote: > From Jakub and David's comments it sounds (if I understood correctly), > you'd like to tie the dma-buf bind/unbind functions to the lifetime of > a netlink socket, rather than a struct file like I was thinking. That > does sound cleaner, but I'm not sure how. Can you link me to any > existing code examples? Or rough pointers to any existing code? I don't have a strong preference whether the lifetime is bound to the socket or not. My main point was that if we're binding lifetimes to processes, it should be done via netlink sockets, not special- -purpose FDs. Inevitably more commands and info will be needed and we'll start reinventing the uAPI wheel which is Netlink. Currently adding state to netlink sockets is a bit raw. You can create an Xarray which stores the per socket state using socket's portid (genl_info->snd_portid) and use netlink_register_notifier() to get notifications when sockets are closed.
On Wed, Jul 19, 2023 at 10:57:11AM -0700, Stephen Hemminger wrote: > Naive idea. > Would it be possible for process to use mmap() on the GPU memory and then > do zero copy TCP receive some how? Or is this what is being proposed. It could be possible, but currently there is no API to recover the underlying dmabuf from the VMA backing the mmap. Also you can't just take arbitary struct pages from any old VMA and make them "netmem" Jason
Am 20.07.23 um 01:24 schrieb Jason Gunthorpe: > On Wed, Jul 19, 2023 at 10:57:11AM -0700, Stephen Hemminger wrote: > >> Naive idea. >> Would it be possible for process to use mmap() on the GPU memory and then >> do zero copy TCP receive some how? Or is this what is being proposed. > It could be possible, but currently there is no API to recover the > underlying dmabuf from the VMA backing the mmap. Sorry for being a bit late, have been on vacation. Well actually this was discussed before to work around problems with Windows applications through wine/proton. Not 100% sure what the outcome of that was, but if I'm not completely mistaken getting the fd behind a VMA should be possible. It might just not be the DMA-buf fd, because we use mmap() re-routing to be able to work around problems with the reverse tracking of mappings. Christian. > > Also you can't just take arbitary struct pages from any old VMA and > make them "netmem" > > Jason > _______________________________________________ > Linaro-mm-sig mailing list -- linaro-mm-sig@lists.linaro.org > To unsubscribe send an email to linaro-mm-sig-leave@lists.linaro.org