| Message ID | 20241210093408.105867-1-dlemoal@kernel.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | NVMe PCI endpoint target driver | expand |
FYI, this looks good to me:
Reviewed-by: Christoph Hellwig <hch@lst.de>
But I've closely worked with Damaien on this, so a look from someone
who is not too close to the work would be useful as well.
On 12/12/24 15:06, Christoph Hellwig wrote: > FYI, this looks good to me: > > Reviewed-by: Christoph Hellwig <hch@lst.de> Thanks. The kernel test robot signaled a build failure that I need to fix (not a big deal) and I have some typo in the documentation too that need fixing. I will post v4 with these fixes. > > But I've closely worked with Damaien on this, so a look from someone > who is not too close to the work would be useful as well. >
Hello,
I have tested the NVMe endpoint function extensively on two extra
platforms to run on, the FriendlyElec CM3588+NAS Kit [1] and NanoPC T6
[2] (with PCIe x1, x2, and x4 links).
Besides testing with Linux based hosts (Ubuntu 20.04, 23.10, 24.04), I
also tested that the NVMe device is recognized by a Windows 10
Enterprise host and can be formatted and used as a regular disk. I
also tested on MacOS Monterey on a Macbook Pro 2016 (Intel based) host
through a M.2 to USB-C adapter [3], the drive is recognized and
usable.
The USB-C adapter is based on the ASM2464PD chipset [4] which does
USB4/Thunderbolt to PCIe/NVMe, I tested with PCI over Thunderbolt with
the MacOS host, so the host sees the NVMe endpoint function directly
as a PCI device and the NVMe drive is seen as such. This works well.
The only test case that did not work is when I tested the ASM2464PD
chipset NVMe to regular USB functionality, where the chipset is the
host, and presents itself to the PC as a "usb-storage" class device,
but this didn't work because the ASM2464PD never enabled the NVMe
controller (CC.EN bit in BAR0), the PCI Link between the ASM2464PD and
endpoint function gets up however, and to the host PC the USB device
is recognized ("usb-storage" class, like a USB stick, e.g., /dev/sda),
but it cannot be read (shows as 0B block device). As I cannot debug
the chipset itself I don't know why the NVMe endpoint doesn't get
enabled. This might very well be a quirk in the ASM2464PD chipset and
is a very specific use case so I don't think it indicates any major
issues with the endpoint function, but I report it here for the sake
of completion.
I have tested with different storage backend devices for actual
storage (USB, eMMC, and NVMe (PCIe x1, x2) on the NAS kit).
In summary, over PCI the endpoint function works well with all three
Linux/MacOS/Windows hosts.
Tested-by: Rick Wertenbroek <rick.wertenbroek@gmail.com>
(I don't know if this applies as I co-developed the endpoint function
with Damien)
Best regards,
Rick
[1] https://wiki.friendlyelec.com/wiki/index.php/CM3588
[2] https://wiki.friendlyelec.com/wiki/index.php/NanoPC-T6
[3] https://www.aliexpress.com/item/1005006316029054.html
[4] https://www.asmedia.com.tw/product/802zX91Yw3tsFgm4/C64ZX59yu4sY1GW5/ASM2464PD
On 12/12/24 19:16, Rick Wertenbroek wrote: > Hello, > I have tested the NVMe endpoint function extensively on two extra > platforms to run on, the FriendlyElec CM3588+NAS Kit [1] and NanoPC T6 > [2] (with PCIe x1, x2, and x4 links). > > Besides testing with Linux based hosts (Ubuntu 20.04, 23.10, 24.04), I > also tested that the NVMe device is recognized by a Windows 10 > Enterprise host and can be formatted and used as a regular disk. I > also tested on MacOS Monterey on a Macbook Pro 2016 (Intel based) host > through a M.2 to USB-C adapter [3], the drive is recognized and > usable. > > The USB-C adapter is based on the ASM2464PD chipset [4] which does > USB4/Thunderbolt to PCIe/NVMe, I tested with PCI over Thunderbolt with > the MacOS host, so the host sees the NVMe endpoint function directly > as a PCI device and the NVMe drive is seen as such. This works well. > > The only test case that did not work is when I tested the ASM2464PD > chipset NVMe to regular USB functionality, where the chipset is the > host, and presents itself to the PC as a "usb-storage" class device, > but this didn't work because the ASM2464PD never enabled the NVMe > controller (CC.EN bit in BAR0), the PCI Link between the ASM2464PD and > endpoint function gets up however, and to the host PC the USB device > is recognized ("usb-storage" class, like a USB stick, e.g., /dev/sda), > but it cannot be read (shows as 0B block device). As I cannot debug > the chipset itself I don't know why the NVMe endpoint doesn't get > enabled. This might very well be a quirk in the ASM2464PD chipset and > is a very specific use case so I don't think it indicates any major > issues with the endpoint function, but I report it here for the sake > of completion. > > I have tested with different storage backend devices for actual > storage (USB, eMMC, and NVMe (PCIe x1, x2) on the NAS kit). > > In summary, over PCI the endpoint function works well with all three > Linux/MacOS/Windows hosts. > > Tested-by: Rick Wertenbroek <rick.wertenbroek@gmail.com> > (I don't know if this applies as I co-developed the endpoint function > with Damien) I think it does apply since I only tested with Linux hosts. Thanks a lot for all the extra testing ! Sending v4 with the build failure fixed and doc typos fixed. > > Best regards, > Rick > > [1] https://wiki.friendlyelec.com/wiki/index.php/CM3588 > [2] https://wiki.friendlyelec.com/wiki/index.php/NanoPC-T6 > [3] https://www.aliexpress.com/item/1005006316029054.html > [4] https://www.asmedia.com.tw/product/802zX91Yw3tsFgm4/C64ZX59yu4sY1GW5/ASM2464PD >
(Note: follow up to v2 of "NVMe PCI endpoint function driver" patch series). This patch series implements an NVMe target driver for the PCI transport using the PCI endpoint framework. The first 4 patches of this series move and cleanup some nvme code that will be reused in following patches. Patch 5 introduces the PCI transport type to allow setting up ports for the new PCI target controller driver. Patch 6 to 9 are improvements of the target core code to allow creating the PCI controller and processing its nvme commands without the need to rely on fabrics commands like the connect command to create the admin and I/O queues. Patch 10 relaxes the SGL check in nvmet_req_init() to allow for PCI admin commands (which must use PRPs). Patches 11 to 15 improve the set/get feature support of the target code to get closer to achieving NVMe specification compliance. These patches though do not implement support for some mandatory features. Patch 16 is the main patch which introduces the NVMe PCI endpoint target driver. This patch commit message provides and overview of the driver design and operation. Finally, patch 17 documents the NVMe PCI endpoint target driver and provides a user guide explaning how to setup an NVMe PCI endpoint device. The patches are base on Linus 6.13-rc2 tree. This driver has been extensively tested using a Radxa Rock5B board (RK3588 Arm SoC). Some tests have also been done using a Pine Rockpro64 board. However, this board does not support DMA channels for the PCI endpoint controller, leading to very poor performance. Using the Radxa Rock5b board and setting up a 4 queue-pairs controller with a null-blk block device loop target, performance was measured using fio as follows: +----------------------------------+------------------------+ | Workload | IOPS (BW) | +----------------------------------+------------------------+ | Rand read, 4KB, QD=1, 1 job | 14.3k IOPS | | Rand read, 4KB, QD=32, 1 job | 80.8k IOPS | | Rand read, 4KB, QD=32, 4 jobs | 131k IOPS | | Rand read, 128KB, QD=32, 1 job | 16.7k IOPS (2.18 GB/s) | | Rand read, 128KB, QD=32, 4 jobs | 17.4k IOPS (2.27 GB/s) | | Rand read, 512KB, QD=32, 1 job | 5380 IOPS (2.82 GB/s) | | Rand read, 512KB, QD=32, 4 jobs | 5206 IOPS (2.27 GB/s) | | Rand write, 128KB, QD=32, 1 job | 9617 IOPS (1.26 GB/s) | | Rand write, 128KB, QD=32, 4 jobs | 8405 IOPS (1.10 GB/s) | +----------------------------------+------------------------+ These results use the default MDTS of the NVMe enpoint driver of 512 KB. This driver is not intended for production use but rather to be a playground for learning NVMe and exploring/testing new NVMe features while providing reasonably good performance. Changes from v2: - Changed all preparatory patches before patch 16 to move more NVMe generic code out of the PCI endpoint target driver and into the target core. - Changed patch 16 to use directly a target controller instead of a host controller. Many aspects of the command management and DMA transfer management have also been simplified, leading to higher performance. - Change the documentation patch to match the above changes Changes from v1: - Added review tag to patch 1 - Modified patch 4 to: - Add Rick's copyright notice - Improve admin command handling (set_features command) to handle the number of queues feature (among others) to enable Windows host - Improved SQ and CQ work items handling Damien Le Moal (17): nvmet: Add vendor_id and subsys_vendor_id subsystem attributes nvmet: Export nvmet_update_cc() and nvmet_cc_xxx() helpers nvmet: Introduce nvmet_get_cmd_effects_admin() nvmet: Add drvdata field to struct nvmet_ctrl nvme: Add PCI transport type nvmet: Improve nvmet_alloc_ctrl() interface and implementation nvmet: Introduce nvmet_req_transfer_len() nvmet: Introduce nvmet_sq_create() and nvmet_cq_create() nvmet: Add support for I/O queue management admin commands nvmet: Do not require SGL for PCI target controller commands nvmet: Introduce get/set_feature controller operations nvmet: Implement host identifier set feature support nvmet: Implement interrupt coalescing feature support nvmet: Implement interrupt config feature support nvmet: Implement arbitration feature support nvmet: New NVMe PCI endpoint target driver Documentation: Document the NVMe PCI endpoint target driver Documentation/PCI/endpoint/index.rst | 1 + .../PCI/endpoint/pci-nvme-function.rst | 14 + Documentation/nvme/index.rst | 12 + .../nvme/nvme-pci-endpoint-target.rst | 365 +++ Documentation/subsystem-apis.rst | 1 + drivers/nvme/target/Kconfig | 10 + drivers/nvme/target/Makefile | 2 + drivers/nvme/target/admin-cmd.c | 388 ++- drivers/nvme/target/configfs.c | 49 + drivers/nvme/target/core.c | 266 +- drivers/nvme/target/discovery.c | 17 + drivers/nvme/target/fabrics-cmd-auth.c | 14 +- drivers/nvme/target/fabrics-cmd.c | 101 +- drivers/nvme/target/nvmet.h | 110 +- drivers/nvme/target/pci-ep.c | 2627 +++++++++++++++++ include/linux/nvme.h | 2 + 16 files changed, 3858 insertions(+), 121 deletions(-) create mode 100644 Documentation/PCI/endpoint/pci-nvme-function.rst create mode 100644 Documentation/nvme/index.rst create mode 100644 Documentation/nvme/nvme-pci-endpoint-target.rst create mode 100644 drivers/nvme/target/pci-ep.c