| Message ID | 20240116-ml-topic-u9p-v1-0-ad8c306f9a4e@pengutronix.de (mailing list archive) |
|---|---|
| Headers | show |
| Series | usb: gadget: 9pfs transport | expand |
Michael Grzeschik wrote on Tue, Jan 16, 2024 at 02:49:40AM +0100: > This series is adding support to mount 9pfs exported filesystems via the > usb gadget interface. It also includes tools and descriptions on how to > translate an tcp 9pfs and use it via the usb interface. So I didn't have time to look at everything through, just want to make sure, this series allows sharing data from an usb gadget (e.g. some device with storage) over 9p as an alternative to things like MTP ? I don't quite understand what the forwarder and diod have to do with this; you're emulating a fake usb device with the forwarder that just transmits requests to diod as backend implementation? But 'usb.core.find(idVendor=0x1D6B, idProduct=0x0109)' looks like it's searching for a real device not creating one, so that doesn't seem to match up... If you have any background information on where you're coming from and where this is headed it'd be great to include in the cover letter. While I had a quick look I'll spare you a second mail for the first patch: Michael Grzeschik wrote on Tue, Jan 16, 2024 at 02:49:41AM +0100: > +static struct p9_trans_module p9_usbg_trans = { > + .name = "usbg", > + .create = p9_usbg_create, > + .close = p9_usbg_close, > + .request = p9_usbg_request, > + .cancel = p9_usbg_cancel, > + .owner = THIS_MODULE, > +}; This is missing a MODULE_ALIAS_9P("usbg") if you want the module to auto-load on `mount -t trans=usbg` -- assuming this can build as a module. I'm also a bit worried that this net/9p-centric code is now also split with drivers/usb/gadget/function/f_9pfs.c and I'll bet you the build will break once in a while when we update global 9p client.c or similar -- I'd be more comfortable having a net/9p/trans_usbg.c or equivalent if possible. Is there a reason this has to be in the usb gadget tree? (Well, I assume from the usb gadget point of view, it's reasonable to similarily prefer this code to stay close to drivers/usb/gadget..) Thanks,
On Tue, 2024-01-16 at 20:45 +0900, Dominique Martinet wrote: > Michael Grzeschik wrote on Tue, Jan 16, 2024 at 02:49:40AM +0100: > > This series is adding support to mount 9pfs exported filesystems via the > > usb gadget interface. It also includes tools and descriptions on how to > > translate an tcp 9pfs and use it via the usb interface. > > So I didn't have time to look at everything through, just want to make > sure, this series allows sharing data from an usb gadget (e.g. some > device with storage) over 9p as an alternative to things like MTP ? It's the other way around. :) The USB host exports a filesystem, while the gadget on the USB device side makes it mountable. Our main use-case is to use it as an alternative to NFS root booting during the development of embedded Linux devices. NFS root works in many cases, but has some downsides, which make it cumbersome to use in more and more cases. NFS root needs correctly configured Ethernet interfaces on both the development host and the target device. On the target, this can interfere with the network configuration that is used for the normal device operation (DHCP client, ...). For the host, configuring a NFS (and perhaps DHCP) server can be an obstacle. For target devices which don't have a real Ethernet interface, NFS root would also work with the USB Ethernet gadget, but this increases the complexity further. As many embedded boards have a USB device port anyway, which is used during development for uploading the boot-loader and to flash filesystem images (i.e. via the fastboot protocol), we want to just reuse that single data cable to allow access to the root filesystem as well. Compared to flashing images, using a network filesystem like NFS and 9P reduces the time between compiling on the host and running the binary on the target, as no flash and reboot cycle is needed. That can get rid of many minutes of waiting over a day. :) > I don't quite understand what the forwarder and diod have to do with > this; you're emulating a fake usb device with the forwarder that just > transmits requests to diod as backend implementation? > But 'usb.core.find(idVendor=0x1D6B, idProduct=0x0109)' looks like it's > searching for a real device not creating one, so that doesn't seem to > match up... diod (9pfs server) and the forwarder are on the development host, where the root filesystem is actually stored. The gadget is initialized during boot (or later) on the embedded board. Then the forwarder will find it on the USB bus and start forwarding requests. It may seem a bit unusual that in this case the requests come from the device and are handled by the host. The reason is that USB device ports are normally not available on PCs, so a connection in the other direction would not work. In the future, the functionality of the forwarder could be integrated into the 9pfs server. Alternatively, an improved forwarder could also react to udev events of gadgets showing up and forward them to different 9PFS server over the network (when you have multiple target devices connected to one USB host). Perhaps, the inverse setup (9PFS server on the USB gadget side, mounted on a PC) also would be useful in the future and could share some of this code. Then, you'd have an alternative to MTP. > If you have any background information on where you're coming from and > where this is headed it'd be great to include in the cover letter. Yes. Probably also in Documentation/. Thanks, Jan
Jan Lübbe wrote on Tue, Jan 16, 2024 at 04:51:41PM +0100: > > So I didn't have time to look at everything through, just want to make > > sure, this series allows sharing data from an usb gadget (e.g. some > > device with storage) over 9p as an alternative to things like MTP ? > > It's the other way around. :) The USB host exports a filesystem, while the > gadget on the USB device side makes it mountable. Our main use-case is to use it > as an alternative to NFS root booting during the development of embedded Linux > devices. NFS root works in many cases, but has some downsides, which make it > cumbersome to use in more and more cases. Oh! Okay, this makes a lot more sense. And that'll need a bit more explanations in the commits & Documentation/ as you've concluded :) > NFS root needs correctly configured Ethernet interfaces on both the development > host and the target device. On the target, this can interfere with the network > configuration that is used for the normal device operation (DHCP client, ...). > For the host, configuring a NFS (and perhaps DHCP) server can be an obstacle. > > For target devices which don't have a real Ethernet interface, NFS root would > also work with the USB Ethernet gadget, but this increases the complexity > further. > > As many embedded boards have a USB device port anyway, which is used during > development for uploading the boot-loader and to flash filesystem images (i.e. > via the fastboot protocol), we want to just reuse that single data cable to > allow access to the root filesystem as well. > > Compared to flashing images, using a network filesystem like NFS and 9P reduces > the time between compiling on the host and running the binary on the target, as > no flash and reboot cycle is needed. That can get rid of many minutes of waiting > over a day. :) My other hat is on embedded development (dayjob at Atmark Techno[1], the only english page linked is about 4 years out of date but I guess it's better than no page at all), so I can understand where you're coming from -- thanks for the background. [1] https://www.atmark-techno.com/english That means I'll actually want to test this, but kind of always busy so it might take a few weeks... Or better, do you happen to know if qemu can create a USB controller that supports OTG so it'll be easy to test for folks with no such hardware? We've got enough 9p protocols that aren't actually tested on a regular basis, it'd be great if we could have something that can run anywhere. > diod (9pfs server) and the forwarder are on the development host, where the root > filesystem is actually stored. The gadget is initialized during boot (or later) > on the embedded board. Then the forwarder will find it on the USB bus and start > forwarding requests. > > It may seem a bit unusual that in this case the requests come from the device > and are handled by the host. The reason is that USB device ports are normally > not available on PCs, so a connection in the other direction would not work. Right, most host PCs won't have OTG available... I was also perplexed by the linux foundation (0x1d6b):0x0109 id, that might be clearer once it's properly documented -- I'll let someone from the usb side chime on this as I have no idea what's appropriate. > In the future, the functionality of the forwarder could be integrated into the > 9pfs server. Alternatively, an improved forwarder could also react to udev > events of gadgets showing up and forward them to different 9PFS server over the > network (when you have multiple target devices connected to one USB host). Plenty of potential work ahead :) Frankly at this stage I don't think it's much simpler than e.g. CDC ethernet gadget and mounting nfs over tcp, but with further improvements it can definitely get simpler. > Perhaps, the inverse setup (9PFS server on the USB gadget side, mounted on a PC) > also would be useful in the future and could share some of this code. Then, > you'd have an alternative to MTP. (Yeah, I'm not actively looking for that -- was just asking because MTP has been kind of dead lately and I'm not aware of any potential alternative, but I didn't go looking for them either -- let's leave that to later)
Hi, W dniu 17.01.2024 o 11:54, Dominique Martinet pisze: > Jan Lübbe wrote on Tue, Jan 16, 2024 at 04:51:41PM +0100: >>> So I didn't have time to look at everything through, just want to make >>> sure, this series allows sharing data from an usb gadget (e.g. some >>> device with storage) over 9p as an alternative to things like MTP ? >> >> It's the other way around. :) The USB host exports a filesystem, while the >> gadget on the USB device side makes it mountable. Our main use-case is to use it >> as an alternative to NFS root booting during the development of embedded Linux >> devices. NFS root works in many cases, but has some downsides, which make it >> cumbersome to use in more and more cases. > > Oh! > Okay, this makes a lot more sense. And that'll need a bit more > explanations in the commits & Documentation/ as you've concluded :) > > >> NFS root needs correctly configured Ethernet interfaces on both the development >> host and the target device. On the target, this can interfere with the network >> configuration that is used for the normal device operation (DHCP client, ...). >> For the host, configuring a NFS (and perhaps DHCP) server can be an obstacle. >> >> For target devices which don't have a real Ethernet interface, NFS root would >> also work with the USB Ethernet gadget, but this increases the complexity >> further. >> >> As many embedded boards have a USB device port anyway, which is used during >> development for uploading the boot-loader and to flash filesystem images (i.e. >> via the fastboot protocol), we want to just reuse that single data cable to >> allow access to the root filesystem as well. >> >> Compared to flashing images, using a network filesystem like NFS and 9P reduces >> the time between compiling on the host and running the binary on the target, as >> no flash and reboot cycle is needed. That can get rid of many minutes of waiting >> over a day. :) > > My other hat is on embedded development (dayjob at Atmark Techno[1], the > only english page linked is about 4 years out of date but I guess it's > better than no page at all), so I can understand where you're coming > from -- thanks for the background. > > [1] https://www.atmark-techno.com/english > > That means I'll actually want to test this, but kind of always busy so > it might take a few weeks... > Or better, do you happen to know if qemu can create a USB controller > that supports OTG so it'll be easy to test for folks with no such > hardware? Maybe dummy_hcd is what you want? Regards, Andrzej > We've got enough 9p protocols that aren't actually tested on a regular > basis, it'd be great if we could have something that can run anywhere. > > >> diod (9pfs server) and the forwarder are on the development host, where the root >> filesystem is actually stored. The gadget is initialized during boot (or later) >> on the embedded board. Then the forwarder will find it on the USB bus and start >> forwarding requests. >> >> It may seem a bit unusual that in this case the requests come from the device >> and are handled by the host. The reason is that USB device ports are normally >> not available on PCs, so a connection in the other direction would not work. > > Right, most host PCs won't have OTG available... > I was also perplexed by the linux foundation (0x1d6b):0x0109 id, that > might be clearer once it's properly documented -- I'll let someone from > the usb side chime on this as I have no idea what's appropriate. > > >> In the future, the functionality of the forwarder could be integrated into the >> 9pfs server. Alternatively, an improved forwarder could also react to udev >> events of gadgets showing up and forward them to different 9PFS server over the >> network (when you have multiple target devices connected to one USB host). > > Plenty of potential work ahead :) > Frankly at this stage I don't think it's much simpler than e.g. CDC > ethernet gadget and mounting nfs over tcp, but with further improvements > it can definitely get simpler. > > >> Perhaps, the inverse setup (9PFS server on the USB gadget side, mounted on a PC) >> also would be useful in the future and could share some of this code. Then, >> you'd have an alternative to MTP. > > (Yeah, I'm not actively looking for that -- was just asking because MTP > has been kind of dead lately and I'm not aware of any potential > alternative, but I didn't go looking for them either -- let's leave that > to later) >
On Fri, Jan 26, 2024 at 08:47:22PM +0100, Andrzej Pietrasiewicz wrote: >Hi, > >W dniu 17.01.2024 o 11:54, Dominique Martinet pisze: >>Jan Lübbe wrote on Tue, Jan 16, 2024 at 04:51:41PM +0100: >>>>So I didn't have time to look at everything through, just want to make >>>>sure, this series allows sharing data from an usb gadget (e.g. some >>>>device with storage) over 9p as an alternative to things like MTP ? >>> >>>It's the other way around. :) The USB host exports a filesystem, while the >>>gadget on the USB device side makes it mountable. Our main use-case is to use it >>>as an alternative to NFS root booting during the development of embedded Linux >>>devices. NFS root works in many cases, but has some downsides, which make it >>>cumbersome to use in more and more cases. >> >>Oh! >>Okay, this makes a lot more sense. And that'll need a bit more >>explanations in the commits & Documentation/ as you've concluded :) >> >> >>>NFS root needs correctly configured Ethernet interfaces on both the development >>>host and the target device. On the target, this can interfere with the network >>>configuration that is used for the normal device operation (DHCP client, ...). >>>For the host, configuring a NFS (and perhaps DHCP) server can be an obstacle. >>> >>>For target devices which don't have a real Ethernet interface, NFS root would >>>also work with the USB Ethernet gadget, but this increases the complexity >>>further. >>> >>>As many embedded boards have a USB device port anyway, which is used during >>>development for uploading the boot-loader and to flash filesystem images (i.e. >>>via the fastboot protocol), we want to just reuse that single data cable to >>>allow access to the root filesystem as well. >>> >>>Compared to flashing images, using a network filesystem like NFS and 9P reduces >>>the time between compiling on the host and running the binary on the target, as >>>no flash and reboot cycle is needed. That can get rid of many minutes of waiting >>>over a day. :) >> >>My other hat is on embedded development (dayjob at Atmark Techno[1], the >>only english page linked is about 4 years out of date but I guess it's >>better than no page at all), so I can understand where you're coming >>from -- thanks for the background. >> >>[1] https://www.atmark-techno.com/english >> >>That means I'll actually want to test this, but kind of always busy so >>it might take a few weeks... >>Or better, do you happen to know if qemu can create a USB controller >>that supports OTG so it'll be easy to test for folks with no such >>hardware? > >Maybe dummy_hcd is what you want? I did a lot of testing with dummy_hcd. So this should work. But of course testing the special case of rootfs is tricky. Since you will have to share the gadget with qemu to boot the rootfs from somewhere else. Regards, Michael >>We've got enough 9p protocols that aren't actually tested on a regular >>basis, it'd be great if we could have something that can run anywhere. >> >> >>>diod (9pfs server) and the forwarder are on the development host, where the root >>>filesystem is actually stored. The gadget is initialized during boot (or later) >>>on the embedded board. Then the forwarder will find it on the USB bus and start >>>forwarding requests. >>> >>>It may seem a bit unusual that in this case the requests come from the device >>>and are handled by the host. The reason is that USB device ports are normally >>>not available on PCs, so a connection in the other direction would not work. >> >>Right, most host PCs won't have OTG available... >>I was also perplexed by the linux foundation (0x1d6b):0x0109 id, that >>might be clearer once it's properly documented -- I'll let someone from >>the usb side chime on this as I have no idea what's appropriate. >> >> >>>In the future, the functionality of the forwarder could be integrated into the >>>9pfs server. Alternatively, an improved forwarder could also react to udev >>>events of gadgets showing up and forward them to different 9PFS server over the >>>network (when you have multiple target devices connected to one USB host). >> >>Plenty of potential work ahead :) >>Frankly at this stage I don't think it's much simpler than e.g. CDC >>ethernet gadget and mounting nfs over tcp, but with further improvements >>it can definitely get simpler. >> >> >>>Perhaps, the inverse setup (9PFS server on the USB gadget side, mounted on a PC) >>>also would be useful in the future and could share some of this code. Then, >>>you'd have an alternative to MTP. >> >>(Yeah, I'm not actively looking for that -- was just asking because MTP >>has been kind of dead lately and I'm not aware of any potential >>alternative, but I didn't go looking for them either -- let's leave that >>to later) >> > >
This series is adding support to mount 9pfs exported filesystems via the usb gadget interface. It also includes tools and descriptions on how to translate an tcp 9pfs and use it via the usb interface. Signed-off-by: Michael Grzeschik <m.grzeschik@pengutronix.de> --- Michael Grzeschik (3): usb: gadget: function: 9pfs usb: gadget: legacy: add 9pfs multi gadget tools: usb: p9_fwd: add usb gadget packet forwarder script Documentation/filesystems/9p.rst | 12 + drivers/usb/gadget/Kconfig | 11 + drivers/usb/gadget/function/Makefile | 2 + drivers/usb/gadget/function/f_9pfs.c | 849 +++++++++++++++++++++++++++++++++++ drivers/usb/gadget/legacy/9pfs.c | 260 +++++++++++ drivers/usb/gadget/legacy/Kconfig | 15 + drivers/usb/gadget/legacy/Makefile | 2 + tools/usb/p9_fwd.py | 194 ++++++++ 8 files changed, 1345 insertions(+) --- base-commit: 052d534373b7ed33712a63d5e17b2b6cdbce84fd change-id: 20240116-ml-topic-u9p-895274530eb1 Best regards,