| Message ID | 20241216141818.111255-1-gustavo.romero@linaro.org (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add ivshmem-flat device | expand |
Gustavo Romero <gustavo.romero@linaro.org> writes: > This is a resend of the series: > > https://lore.kernel.org/qemu-devel/20240222222218.2261956-1-gustavo.romero@linaro.org/ > > rebased on the current master. The series was sent about 9 months ago and > remains relevant. Besides addressing the longstanding issue: > > https://gitlab.com/qemu-project/qemu/-/issues/1134 > > it has generated interest in the community at least twice since its last > version, from different contexts: > > https://lists.nongnu.org/archive/html/qemu-discuss/2024-05/msg00003.html > https://lists.nongnu.org/archive/html/qemu-devel/2024-09/msg00374.html > > This suggests the series is being used out-of-tree in various contexts, such > as experiments with heterogeneous architectures. > > But due to the fact it relies on sysbus, which is marked for future removal, > some maintainers objected to accepting the patchset, causing it to be held in > the ML. Actually, I inquired about the use cases, and was told it's for OpenAMP. I challenged the use of ivshmem for that purpose in some detail[*], but got no reply. > However, given the ongoing community interest and since currently there > isn't a better way on QEMU than using sysbus for the wiring needs of this > device (e.g. to wire the device to a CPU IRQ input line), I'd kindly like to ask > maintainers to reconsider its acceptance. [*] https://lore.kernel.org/qemu-devel/87zfth4psf.fsf@pond.sub.org/ [...]
Markus Armbruster <armbru@redhat.com> writes: > Gustavo Romero <gustavo.romero@linaro.org> writes: > >> This is a resend of the series: >> >> https://lore.kernel.org/qemu-devel/20240222222218.2261956-1-gustavo.romero@linaro.org/ >> >> rebased on the current master. The series was sent about 9 months ago and >> remains relevant. Besides addressing the longstanding issue: >> >> https://gitlab.com/qemu-project/qemu/-/issues/1134 >> >> it has generated interest in the community at least twice since its last >> version, from different contexts: >> >> https://lists.nongnu.org/archive/html/qemu-discuss/2024-05/msg00003.html >> https://lists.nongnu.org/archive/html/qemu-devel/2024-09/msg00374.html >> >> This suggests the series is being used out-of-tree in various contexts, such >> as experiments with heterogeneous architectures. >> >> But due to the fact it relies on sysbus, which is marked for future removal, >> some maintainers objected to accepting the patchset, causing it to be held in >> the ML. > > Actually, I inquired about the use cases, and was told it's for OpenAMP. > I challenged the use of ivshmem for that purpose in some detail[*], but > got no reply. > >> However, given the ongoing community interest and since currently there >> isn't a better way on QEMU than using sysbus for the wiring needs of this >> device (e.g. to wire the device to a CPU IRQ input line), I'd kindly like to ask >> maintainers to reconsider its acceptance. > > [*] https://lore.kernel.org/qemu-devel/87zfth4psf.fsf@pond.sub.org/ Yes the principle use case is modelling asymmetric set ups with two sets of CPU cores attached only by a piece of shared RAM with maybe a signalling an IRQ. As the CPUs are doing the work we want to test both sides (VirtIO device and driver) rather than provide an emulation. Once we have a single QEMU binary that is dynamically configurable and supports heterogeneous setups then we can model it within QEMU itself. However until that point 2 QEMU's and some shared memory is the easiest way to test such things. This is currently being worked on as part of the larger HVAC project: https://linaro.atlassian.net/wiki/spaces/HVAC/overview where we are working on a new VirtIO transport layer (virtio-msg) that doesn't have the issues associated with lack of trapped configuration space. > > [...]
This is a resend of the series: https://lore.kernel.org/qemu-devel/20240222222218.2261956-1-gustavo.romero@linaro.org/ rebased on the current master. The series was sent about 9 months ago and remains relevant. Besides addressing the longstanding issue: https://gitlab.com/qemu-project/qemu/-/issues/1134 it has generated interest in the community at least twice since its last version, from different contexts: https://lists.nongnu.org/archive/html/qemu-discuss/2024-05/msg00003.html https://lists.nongnu.org/archive/html/qemu-devel/2024-09/msg00374.html This suggests the series is being used out-of-tree in various contexts, such as experiments with heterogeneous architectures. But due to the fact it relies on sysbus, which is marked for future removal, some maintainers objected to accepting the patchset, causing it to be held in the ML. However, given the ongoing community interest and since currently there isn't a better way on QEMU than using sysbus for the wiring needs of this device (e.g. to wire the device to a CPU IRQ input line), I'd kindly like to ask maintainers to reconsider its acceptance. I've kept the sysbus wiring part self-contained in patch: "hw/misc/ivshmem-flat: Allow device to wire itself on sysbus" and, as I said in the QEMU/KVM Developers Conference call last week I can commit to updating the API when the time comes to deprecate the sysbus API. In essence, this patchset introduces a new device, named ivshmem-flat, which is similar to the current ivshmem device but does not require a PCI bus. It implements the ivshmem status and control registers as MMRs and the shared memory as a directly accessible memory region in the VM memory layout. It's meant to be used on machines like those with Cortex-M MCUs, which usually lack a PCI bus, e.g., lm3s6965evb and mps2-an385. Additionally, it has the benefit of requiring a tiny 'device driver,' which is helpful on some RTOSes, like Zephyr, that run on memory-constrained resource targets. The patchset includes a QTest for the ivshmem-flat device, which can be run with: $ make -C ./build check-qtest-arm -j 32 provided QEMU was built with support for target "arm-softmmu": $ ../configure --target-list=arm-softmmu [...] Nonetheless, it's also possible to experiment with it in two more interesting ways: (a) using two Cortex-M VMs running Zephyr; or (b) using one aarch64 VM running Linux with the ivshmem PCI device and another arm (Cortex-M) VM running Zephyr with the new ivshmem-flat device. To experiment with (a), clone this Zephyr repo [0], set the Zephyr build environment [1], and follow the instructions in the 'ivshmem' sample main.c [2]. [0] https://github.com/gromero/zephyr/tree/ivshmem [1] https://docs.zephyrproject.org/latest/develop/getting_started/index.html [2] https://github.com/gromero/zephyr/commit/73fbd481e352b25ae5483ba5048a2182b90b7f00#diff-16fa1f481a49b995d0d1a62da37b9f33033f5ee477035e73465e7208521ddbe0R9-R70 [3] https://lore.kernel.org/qemu-devel/20231127052024.435743-1-gustavo.romero@linaro.org/ To experiment with (b): $ git clone -b uio_ivshmem --single-branch https://github.com/gromero/linux.git $ cd linux $ wget https://people.linaro.org/~gustavo.romero/ivshmem/arm64_uio_ivshmem.config -O .config If in an x86_64 machine, cross compile the kernel, for instance: $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j 36 Install image in some directory, let's say, in ~/linux: $ mkdir ~/linux $ export INSTALL_PATH=~/linux $ make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- -j 36 install or, if you prefer, download the compiled image from: $ wget https://people.linaro.org/~gustavo.romero/ivshmem/vmlinuz-6.6.0-rc1-g28f3f88ee261 ... and then the rootfs: $ wget https://people.linaro.org/~gustavo.romero/ivshmem/rootfs.qcow2 Now, build QEMU with this patchset applied: $ mkdir build && cd build $ ../configure --target-list=arm-softmmu,aarch64-softmmu $ make -j 36 Start the ivshmem server: $ contrib/ivshmem-server/ivshmem-server -F Start the aarch64 VM + Linux + ivshmem PCI device: $ ./qemu-system-aarch64 -kernel ~/linux/vmlinuz-6.6.0-rc1-g28f3f88ee261 -append "root=/dev/vda initrd=/bin/bash console=ttyAMA0,115200" -drive file=~/linux/rootfs.qcow2,media=disk,if=virtio -machine virt-6.2 -nographic -accel tcg -cpu cortex-a57 -m 8192 -netdev bridge,id=hostnet0,br=virbr0,helper=/usr/lib/qemu/qemu-bridge-helper -device pcie-root-port,port=8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:d9:d1:12,bus=pci.1,addr=0x0 -device ivshmem-doorbell,vectors=2,chardev=ivshmem -chardev socket,path=/tmp/ivshmem_socket,id=ivshmem Log into the VM with user/pass: root/abc123 should show: [ 2.656367] uio_ivshmem 0000:00:02.0: ivshmem-mmr at 0x0000000010203000, size 0x0000000000001000 [ 2.656931] uio_ivshmem 0000:00:02.0: ivshmem-shmem at 0x0000008000000000, size 0x0000000000400000 [ 2.662554] uio_ivshmem 0000:00:02.0: module successfully loaded In another console, clone and build Zephyr image from 'uio_ivhsmem' branch: $ git clone -b uio_ivshmem --single-branch https://github.com/gromero/zephyr $ west -v --verbose build -p always -b qemu_cortex_m3 ./samples/uio_ivshmem/ ... and then start the arm VM + Zephyr image + ivshmem-flat device: $ ./qemu-system-arm -machine lm3s6965evb -nographic -net none -chardev socket,path=/tmp/ivshmem_socket,id=ivshmem_flat -device ivshmem-flat,chardev=ivshmem_flat,x-irq-qompath='/machine/unattached/device[1]/nvic/unnamed-gpio-in[0]',x-bus-qompath='/sysbus' -kernel ~/zephyrproject/zephyr/build/qemu_cortex_m3/uio_ivshmem/zephyr/zephyr.elf You should see something like: *** Booting Zephyr OS build zephyr-v3.3.0-8350-gfb003e583600 *** *** Board: qemu_cortex_m3 *** Installing direct IRQ handler for external IRQ0 (Exception #16)... *** Enabling IRQ0 in the NVIC logic... *** Received IVSHMEM PEER ID: 7 *** Waiting notification from peers to start... Now, from the Linux terminal, notify the arm VM from the aarch64 VM (use the "IVSHMEM PEER ID" reported by Zephyr as the third argument, in this example: 7): $ ./uio_ivshmem_test /dev/uio0 7 MMRs mapped at 0xffff8fb28000 in VMA. shmem mapped at 0xffff8f728000 in VMA. mmr0: 0 0 mmr1: 0 0 mmr2: 6 6 mmr3: 0 0 Data ok. 4194304 byte(s) checked. The arm VM should report something like: *** Got interrupt at vector 0! *** Writting constant 0xb5b5b5b5 to shmem... done! *** Notifying back peer ID 6 at vector 0... -- v2: - Correct code style - Correct trace event format strings - Include minimum headers in ivshmem-flat.h - Allow ivshmem_flat_recv_msg() take NULL - Factored ivshmem_flat_connect_server() out - Split sysbus-auto-wire controversial code in different patch - Document QDev interface v3: - Addressed all comments from Thomas Huth about qtest: 1) Use of g_usleep + number of attemps for timeout 2) Use of g_get_tmp_dir instead of hard-coded /tmp 3) Test if machine lm3s6965evb is available, if not skip test - Use of qemu_irq_pulse instead of 2x qemu_set_irq - Fixed all tests for new device options and IRQ name change - Updated doc and commit messages regarding new/deleted device options - Turned device options 'x-bus-address-iomem' and 'x-bus-address-shmem' mandatory Cheers, Gustavo Gustavo Romero (7): hw/misc/ivshmem-flat: Add ivshmem-flat device hw/misc/ivshmem-flat: Allow device to wire itself on sysbus hw/arm: Allow some machines to use the ivshmem-flat device hw/misc/ivshmem: Rename ivshmem to ivshmem-pci tests/qtest: Reorganize common code in ivshmem-test tests/qtest: Add API functions to capture IRQ toggling tests/qtest: Add ivshmem-flat test docs/system/devices/ivshmem-flat.rst | 90 +++++ hw/arm/mps2.c | 3 + hw/arm/stellaris.c | 3 + hw/arm/virt.c | 2 + hw/core/sysbus-fdt.c | 2 + hw/misc/Kconfig | 5 + hw/misc/ivshmem-flat.c | 531 +++++++++++++++++++++++++++ hw/misc/{ivshmem.c => ivshmem-pci.c} | 0 hw/misc/meson.build | 4 +- hw/misc/trace-events | 16 + include/hw/misc/ivshmem-flat.h | 94 +++++ tests/qtest/ivshmem-flat-test.c | 338 +++++++++++++++++ tests/qtest/ivshmem-test.c | 113 +----- tests/qtest/ivshmem-utils.c | 156 ++++++++ tests/qtest/ivshmem-utils.h | 56 +++ tests/qtest/libqtest.c | 24 ++ tests/qtest/libqtest.h | 28 ++ tests/qtest/meson.build | 6 +- 18 files changed, 1362 insertions(+), 109 deletions(-) create mode 100644 docs/system/devices/ivshmem-flat.rst create mode 100644 hw/misc/ivshmem-flat.c rename hw/misc/{ivshmem.c => ivshmem-pci.c} (100%) create mode 100644 include/hw/misc/ivshmem-flat.h create mode 100644 tests/qtest/ivshmem-flat-test.c create mode 100644 tests/qtest/ivshmem-utils.c create mode 100644 tests/qtest/ivshmem-utils.h