| Message ID | 1602780274-29141-1-git-send-email-olekstysh@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | IOREQ feature (+ virtio-mmio) on Arm | expand |
Hi Oleksandr, I would like to try this on my arm64 board. According to your comments in the patch, I made this config file. # cat debian.conf name = "debian" type = "pvh" vcpus = 8 memory = 512 kernel = "/opt/agl/vmlinuz-5.9.0-1-arm64" ramdisk = "/opt/agl/initrd.img-5.9.0-1-arm64" cmdline= "console=hvc0 earlyprintk=xen root=/dev/xvda1 rw" disk = [ '/opt/agl/debian.qcow2,qcow2,hda' ] vif = [ 'mac=00:16:3E:74:3d:76,bridge=xenbr0' ] virtio = 1 vdisk = [ 'backend=Dom0, disks=ro:/dev/sda1' ] And tried to boot a DomU, but I got below error. # xl create -c debian.conf Parsing config from debian.conf libxl: error: libxl_create.c:1863:domcreate_attach_devices: Domain 1:unable to add virtio_disk devices libxl: error: libxl_domain.c:1218:destroy_domid_pci_done: Domain 1:xc_domain_pause failed libxl: error: libxl_dom.c:39:libxl__domain_type: unable to get domain type for domid=1 libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain 1:Unable to destroy guest libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain 1:Destruction of domain failed Could you tell me how can I test it? Thank you, 2020年10月16日(金) 1:46 Oleksandr Tyshchenko <olekstysh@gmail.com>: > > From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> > > Hello all. > > The purpose of this patch series is to add IOREQ/DM support to Xen on Arm. > You can find an initial discussion at [1] and RFC/V1 series at [2]/[3]. > Xen on Arm requires some implementation to forward guest MMIO access to a device > model in order to implement virtio-mmio backend or even mediator outside of hypervisor. > As Xen on x86 already contains required support this series tries to make it common > and introduce Arm specific bits plus some new functionality. Patch series is based on > Julien's PoC "xen/arm: Add support for Guest IO forwarding to a device emulator". > Besides splitting existing IOREQ/DM support and introducing Arm side, the series > also includes virtio-mmio related changes (last 2 patches for toolstack) > for the reviewers to be able to see how the whole picture could look like. > > According to the initial discussion there are a few open questions/concerns > regarding security, performance in VirtIO solution: > 1. virtio-mmio vs virtio-pci, SPI vs MSI, different use-cases require different > transport... > 2. virtio backend is able to access all guest memory, some kind of protection > is needed: 'virtio-iommu in Xen' vs 'pre-shared-memory & memcpys in guest' > 3. interface between toolstack and 'out-of-qemu' virtio backend, avoid using > Xenstore in virtio backend if possible. > 4. a lot of 'foreing mapping' could lead to the memory exhaustion, Julien > has some idea regarding that. > > Looks like all of them are valid and worth considering, but the first thing > which we need on Arm is a mechanism to forward guest IO to a device emulator, > so let's focus on it in the first place. > > *** > > There are a lot of changes since RFC series, almost all TODOs were resolved on Arm, > Arm code was improved and hardened, common IOREQ/DM code became really arch-agnostic > (without HVM-ism), but one TODO still remains which is "PIO handling" on Arm. > The "PIO handling" TODO is expected to left unaddressed for the current series. > It is not an big issue for now while Xen doesn't have support for vPCI on Arm. > On Arm64 they are only used for PCI IO Bar and we would probably want to expose > them to emulator as PIO access to make a DM completely arch-agnostic. So "PIO handling" > should be implemented when we add support for vPCI. > > I left interface untouched in the following patch > "xen/dm: Introduce xendevicemodel_set_irq_level DM op" > since there is still an open discussion what interface to use/what > information to pass to the hypervisor. > > Also I decided to drop the following patch: > "[RFC PATCH V1 07/12] A collection of tweaks to be able to run emulator in driver domain" > as I got an advise to write our own policy using FLASK which would cover our use > case (with emulator in driver domain) rather than tweak Xen. > > There are two patches on review this series depends on (each involved patch in this series > contains this note as well): > 1. https://patchwork.kernel.org/patch/11816689 > 2. https://patchwork.kernel.org/patch/11803383 > > Please note, that IOREQ feature is disabled by default within this series. > > *** > > Patch series [4] was rebased on recent "staging branch" > (8a62dee x86/vLAPIC: don't leak regs page from vlapic_init() upon error) and tested on > Renesas Salvator-X board + H3 ES3.0 SoC (Arm64) with virtio-mmio disk backend (we will > share it later) running in driver domain and unmodified Linux Guest running on existing > virtio-blk driver (frontend). No issues were observed. Guest domain 'reboot/destroy' > use-cases work properly. Patch series was only build-tested on x86. > > Please note, build-test passed for the following modes: > 1. x86: CONFIG_HVM=y / CONFIG_IOREQ_SERVER=y (default) > 2. x86: #CONFIG_HVM is not set / #CONFIG_IOREQ_SERVER is not set > 3. Arm64: CONFIG_HVM=y / CONFIG_IOREQ_SERVER=y > 4. Arm64: CONFIG_HVM=y / #CONFIG_IOREQ_SERVER is not set (default) > 5. Arm32: CONFIG_HVM=y / CONFIG_IOREQ_SERVER=y > 6. Arm32: CONFIG_HVM=y / #CONFIG_IOREQ_SERVER is not set (default) > > *** > > Any feedback/help would be highly appreciated. > > [1] https://lists.xenproject.org/archives/html/xen-devel/2020-07/msg00825.html > [2] https://lists.xenproject.org/archives/html/xen-devel/2020-08/msg00071.html > [3] https://lists.xenproject.org/archives/html/xen-devel/2020-09/msg00732.html > [4] https://github.com/otyshchenko1/xen/commits/ioreq_4.14_ml3 > > Julien Grall (5): > xen/dm: Make x86's DM feature common > xen/mm: Make x86's XENMEM_resource_ioreq_server handling common > arm/ioreq: Introduce arch specific bits for IOREQ/DM features > xen/dm: Introduce xendevicemodel_set_irq_level DM op > libxl: Introduce basic virtio-mmio support on Arm > > Oleksandr Tyshchenko (18): > x86/ioreq: Prepare IOREQ feature for making it common > xen/ioreq: Make x86's IOREQ feature common > xen/ioreq: Make x86's hvm_ioreq_needs_completion() common > xen/ioreq: Provide alias for the handle_mmio() > xen/ioreq: Make x86's hvm_mmio_first(last)_byte() common > xen/ioreq: Make x86's hvm_ioreq_(page/vcpu/server) structs common > xen/ioreq: Move x86's ioreq_gfn(server) to struct domain > xen/ioreq: Introduce ioreq_params to abstract accesses to > arch.hvm.params > xen/ioreq: Move x86's io_completion/io_req fields to struct vcpu > xen/ioreq: Remove "hvm" prefixes from involved function names > xen/ioreq: Use guest_cmpxchg64() instead of cmpxchg() > xen/arm: Stick around in leave_hypervisor_to_guest until I/O has > completed > xen/mm: Handle properly reference in set_foreign_p2m_entry() on Arm > xen/ioreq: Introduce domain_has_ioreq_server() > xen/arm: io: Abstract sign-extension > xen/ioreq: Make x86's send_invalidate_req() common > xen/arm: Add mapcache invalidation handling > [RFC] libxl: Add support for virtio-disk configuration > > MAINTAINERS | 8 +- > tools/libs/devicemodel/core.c | 18 + > tools/libs/devicemodel/include/xendevicemodel.h | 4 + > tools/libs/devicemodel/libxendevicemodel.map | 1 + > tools/libs/light/Makefile | 1 + > tools/libs/light/libxl_arm.c | 94 +- > tools/libs/light/libxl_create.c | 1 + > tools/libs/light/libxl_internal.h | 1 + > tools/libs/light/libxl_types.idl | 16 + > tools/libs/light/libxl_types_internal.idl | 1 + > tools/libs/light/libxl_virtio_disk.c | 109 ++ > tools/xl/Makefile | 2 +- > tools/xl/xl.h | 3 + > tools/xl/xl_cmdtable.c | 15 + > tools/xl/xl_parse.c | 116 ++ > tools/xl/xl_virtio_disk.c | 46 + > xen/arch/arm/Makefile | 2 + > xen/arch/arm/dm.c | 89 ++ > xen/arch/arm/domain.c | 9 + > xen/arch/arm/hvm.c | 4 + > xen/arch/arm/io.c | 29 +- > xen/arch/arm/ioreq.c | 126 ++ > xen/arch/arm/p2m.c | 29 + > xen/arch/arm/traps.c | 58 +- > xen/arch/x86/Kconfig | 1 + > xen/arch/x86/hvm/Makefile | 1 - > xen/arch/x86/hvm/dm.c | 291 +---- > xen/arch/x86/hvm/emulate.c | 60 +- > xen/arch/x86/hvm/hvm.c | 24 +- > xen/arch/x86/hvm/hypercall.c | 9 +- > xen/arch/x86/hvm/intercept.c | 5 +- > xen/arch/x86/hvm/io.c | 26 +- > xen/arch/x86/hvm/ioreq.c | 1533 ----------------------- > xen/arch/x86/hvm/stdvga.c | 10 +- > xen/arch/x86/hvm/svm/nestedsvm.c | 2 +- > xen/arch/x86/hvm/vmx/realmode.c | 6 +- > xen/arch/x86/hvm/vmx/vvmx.c | 2 +- > xen/arch/x86/mm.c | 46 +- > xen/arch/x86/mm/p2m.c | 13 +- > xen/arch/x86/mm/shadow/common.c | 2 +- > xen/common/Kconfig | 3 + > xen/common/Makefile | 2 + > xen/common/dm.c | 292 +++++ > xen/common/ioreq.c | 1443 +++++++++++++++++++++ > xen/common/memory.c | 50 +- > xen/include/asm-arm/domain.h | 5 + > xen/include/asm-arm/hvm/ioreq.h | 109 ++ > xen/include/asm-arm/mm.h | 8 - > xen/include/asm-arm/mmio.h | 1 + > xen/include/asm-arm/p2m.h | 19 +- > xen/include/asm-arm/paging.h | 4 + > xen/include/asm-arm/traps.h | 24 + > xen/include/asm-x86/hvm/domain.h | 50 +- > xen/include/asm-x86/hvm/emulate.h | 2 +- > xen/include/asm-x86/hvm/io.h | 17 - > xen/include/asm-x86/hvm/ioreq.h | 198 ++- > xen/include/asm-x86/hvm/vcpu.h | 18 - > xen/include/asm-x86/mm.h | 4 - > xen/include/asm-x86/p2m.h | 20 +- > xen/include/public/arch-arm.h | 5 + > xen/include/public/hvm/dm_op.h | 16 + > xen/include/xen/dm.h | 44 + > xen/include/xen/ioreq.h | 143 +++ > xen/include/xen/p2m-common.h | 4 + > xen/include/xen/sched.h | 37 + > xen/include/xsm/dummy.h | 4 +- > xen/include/xsm/xsm.h | 6 +- > xen/xsm/dummy.c | 2 +- > xen/xsm/flask/hooks.c | 5 +- > 69 files changed, 3223 insertions(+), 2125 deletions(-) > create mode 100644 tools/libs/light/libxl_virtio_disk.c > create mode 100644 tools/xl/xl_virtio_disk.c > create mode 100644 xen/arch/arm/dm.c > create mode 100644 xen/arch/arm/ioreq.c > delete mode 100644 xen/arch/x86/hvm/ioreq.c > create mode 100644 xen/common/dm.c > create mode 100644 xen/common/ioreq.c > create mode 100644 xen/include/asm-arm/hvm/ioreq.h > create mode 100644 xen/include/xen/dm.h > create mode 100644 xen/include/xen/ioreq.h > > -- > 2.7.4 > >
On Thu, Oct 29, 2020 at 9:42 AM Masami Hiramatsu < masami.hiramatsu@linaro.org> wrote: > Hi Oleksandr, > Hi Masami [sorry for the possible format issue] > > I would like to try this on my arm64 board. > Glad to hear you are interested in this topic. > > According to your comments in the patch, I made this config file. > # cat debian.conf > name = "debian" > type = "pvh" > vcpus = 8 > memory = 512 > kernel = "/opt/agl/vmlinuz-5.9.0-1-arm64" > ramdisk = "/opt/agl/initrd.img-5.9.0-1-arm64" > cmdline= "console=hvc0 earlyprintk=xen root=/dev/xvda1 rw" > disk = [ '/opt/agl/debian.qcow2,qcow2,hda' ] > vif = [ 'mac=00:16:3E:74:3d:76,bridge=xenbr0' ] > virtio = 1 > vdisk = [ 'backend=Dom0, disks=ro:/dev/sda1' ] > > And tried to boot a DomU, but I got below error. > > # xl create -c debian.conf > Parsing config from debian.conf > libxl: error: libxl_create.c:1863:domcreate_attach_devices: Domain > 1:unable to add virtio_disk devices > libxl: error: libxl_domain.c:1218:destroy_domid_pci_done: Domain > 1:xc_domain_pause failed > libxl: error: libxl_dom.c:39:libxl__domain_type: unable to get domain > type for domid=1 > libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain > 1:Unable to destroy guest > libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain > 1:Destruction of domain failed > > > Could you tell me how can I test it? > I assume it is due to the lack of the virtio-disk backend (which I haven't shared yet as I focused on the IOREQ/DM support on Arm in the first place). Could you wait a little bit, I am going to share it soon.
On Thu, 29 Oct 2020, Oleksandr Tyshchenko wrote: > On Thu, Oct 29, 2020 at 9:42 AM Masami Hiramatsu <masami.hiramatsu@linaro.org> wrote: > Hi Oleksandr, > > Hi Masami > > [sorry for the possible format issue] > > > I would like to try this on my arm64 board. > > Glad to hear you are interested in this topic. > > > According to your comments in the patch, I made this config file. > # cat debian.conf > name = "debian" > type = "pvh" > vcpus = 8 > memory = 512 > kernel = "/opt/agl/vmlinuz-5.9.0-1-arm64" > ramdisk = "/opt/agl/initrd.img-5.9.0-1-arm64" > cmdline= "console=hvc0 earlyprintk=xen root=/dev/xvda1 rw" > disk = [ '/opt/agl/debian.qcow2,qcow2,hda' ] > vif = [ 'mac=00:16:3E:74:3d:76,bridge=xenbr0' ] > virtio = 1 > vdisk = [ 'backend=Dom0, disks=ro:/dev/sda1' ] > > And tried to boot a DomU, but I got below error. > > # xl create -c debian.conf > Parsing config from debian.conf > libxl: error: libxl_create.c:1863:domcreate_attach_devices: Domain > 1:unable to add virtio_disk devices > libxl: error: libxl_domain.c:1218:destroy_domid_pci_done: Domain > 1:xc_domain_pause failed > libxl: error: libxl_dom.c:39:libxl__domain_type: unable to get domain > type for domid=1 > libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain > 1:Unable to destroy guest > libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain > 1:Destruction of domain failed > > > Could you tell me how can I test it? > > > I assume it is due to the lack of the virtio-disk backend (which I haven't shared yet as I focused on the IOREQ/DM support on Arm in the > first place). > Could you wait a little bit, I am going to share it soon. Do you have a quick-and-dirty hack you can share in the meantime? Even just on github as a special branch? It would be very useful to be able to have a test-driver for the new feature.
Oleksandr Tyshchenko <olekstysh@gmail.com> writes: > On Thu, Oct 29, 2020 at 9:42 AM Masami Hiramatsu < > masami.hiramatsu@linaro.org> wrote: > >> Hi Oleksandr, >> > Hi Masami > > [sorry for the possible format issue] > > >> >> I would like to try this on my arm64 board. >> > Glad to hear you are interested in this topic. > > >> >> According to your comments in the patch, I made this config file. >> # cat debian.conf >> name = "debian" >> type = "pvh" >> vcpus = 8 >> memory = 512 >> kernel = "/opt/agl/vmlinuz-5.9.0-1-arm64" >> ramdisk = "/opt/agl/initrd.img-5.9.0-1-arm64" >> cmdline= "console=hvc0 earlyprintk=xen root=/dev/xvda1 rw" >> disk = [ '/opt/agl/debian.qcow2,qcow2,hda' ] >> vif = [ 'mac=00:16:3E:74:3d:76,bridge=xenbr0' ] >> virtio = 1 >> vdisk = [ 'backend=Dom0, disks=ro:/dev/sda1' ] >> >> And tried to boot a DomU, but I got below error. >> >> # xl create -c debian.conf >> Parsing config from debian.conf >> libxl: error: libxl_create.c:1863:domcreate_attach_devices: Domain >> 1:unable to add virtio_disk devices >> libxl: error: libxl_domain.c:1218:destroy_domid_pci_done: Domain >> 1:xc_domain_pause failed >> libxl: error: libxl_dom.c:39:libxl__domain_type: unable to get domain >> type for domid=1 >> libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain >> 1:Unable to destroy guest >> libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain >> 1:Destruction of domain failed >> >> >> Could you tell me how can I test it? >> > > I assume it is due to the lack of the virtio-disk backend (which I haven't > shared yet as I focused on the IOREQ/DM support on Arm in the first place). > Could you wait a little bit, I am going to share it soon. I assume this is wiring up the required bits of support to handle the IOREQ requests in QEMU? We are putting together a PoC demo to show a virtio enabled image (AGL) running on both KVM and Xen hypervisors so we are keen to see your code as soon as you can share it. I'm currently preparing a patch series for QEMU which fixes the recent breakage caused by changes to the build system. As part of that I've separated CONFIG_XEN and CONFIG_XEN_HVM so it's possible to build i386-softmmu with unneeded for ARM bits. Hopefully this will allow me to create a qemu-aarch64-system binary with just the PV related models in it. Talking to Stefano it probably makes sense going forward to introduce a new IOREQ aware machine type for QEMU that doesn't bring in the rest of the x86 overhead. I was thinking maybe xenvirt? You've tested with virtio-block but we'd also like to extend this to other arbitrary virtio devices. I guess we will need some sort of mechanism to inform the QEMU command line where each device sits in the virtio-mmio bus so the FDT Xen delivers to the guest matches up with what QEMU is ready to serve requests for?
On Thu, 29 Oct 2020, Alex Bennée wrote: > Oleksandr Tyshchenko <olekstysh@gmail.com> writes: > > On Thu, Oct 29, 2020 at 9:42 AM Masami Hiramatsu < > > masami.hiramatsu@linaro.org> wrote: > > > >> Hi Oleksandr, > >> > > Hi Masami > > > > [sorry for the possible format issue] > > > > > >> > >> I would like to try this on my arm64 board. > >> > > Glad to hear you are interested in this topic. > > > > > >> > >> According to your comments in the patch, I made this config file. > >> # cat debian.conf > >> name = "debian" > >> type = "pvh" > >> vcpus = 8 > >> memory = 512 > >> kernel = "/opt/agl/vmlinuz-5.9.0-1-arm64" > >> ramdisk = "/opt/agl/initrd.img-5.9.0-1-arm64" > >> cmdline= "console=hvc0 earlyprintk=xen root=/dev/xvda1 rw" > >> disk = [ '/opt/agl/debian.qcow2,qcow2,hda' ] > >> vif = [ 'mac=00:16:3E:74:3d:76,bridge=xenbr0' ] > >> virtio = 1 > >> vdisk = [ 'backend=Dom0, disks=ro:/dev/sda1' ] > >> > >> And tried to boot a DomU, but I got below error. > >> > >> # xl create -c debian.conf > >> Parsing config from debian.conf > >> libxl: error: libxl_create.c:1863:domcreate_attach_devices: Domain > >> 1:unable to add virtio_disk devices > >> libxl: error: libxl_domain.c:1218:destroy_domid_pci_done: Domain > >> 1:xc_domain_pause failed > >> libxl: error: libxl_dom.c:39:libxl__domain_type: unable to get domain > >> type for domid=1 > >> libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain > >> 1:Unable to destroy guest > >> libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain > >> 1:Destruction of domain failed > >> > >> > >> Could you tell me how can I test it? > >> > > > > I assume it is due to the lack of the virtio-disk backend (which I haven't > > shared yet as I focused on the IOREQ/DM support on Arm in the first place). > > Could you wait a little bit, I am going to share it soon. > > I assume this is wiring up the required bits of support to handle the > IOREQ requests in QEMU? We are putting together a PoC demo to show > a virtio enabled image (AGL) running on both KVM and Xen hypervisors so > we are keen to see your code as soon as you can share it. > > I'm currently preparing a patch series for QEMU which fixes the recent > breakage caused by changes to the build system. As part of that I've > separated CONFIG_XEN and CONFIG_XEN_HVM so it's possible to build > i386-softmmu with unneeded for ARM bits. Hopefully this will allow me to > create a qemu-aarch64-system binary with just the PV related models in > it. > > Talking to Stefano it probably makes sense going forward to introduce a > new IOREQ aware machine type for QEMU that doesn't bring in the rest of > the x86 overhead. I was thinking maybe xenvirt? > > You've tested with virtio-block but we'd also like to extend this to > other arbitrary virtio devices. I guess we will need some sort of > mechanism to inform the QEMU command line where each device sits in the > virtio-mmio bus so the FDT Xen delivers to the guest matches up with > what QEMU is ready to serve requests for? That would be great. As a reference, given that the domU memory mapping layout is fixed, on x86 we just made sure that QEMU's idea of where the devices are is the same of Xen's. What you are suggesting would make it much more flexible and clearer.
Hi Stefano [sorry for the possible format issue] On Thu, Oct 29, 2020 at 9:53 PM Stefano Stabellini <sstabellini@kernel.org> wrote: > On Thu, 29 Oct 2020, Oleksandr Tyshchenko wrote: > > On Thu, Oct 29, 2020 at 9:42 AM Masami Hiramatsu < > masami.hiramatsu@linaro.org> wrote: > > Hi Oleksandr, > > > > Hi Masami > > > > [sorry for the possible format issue] > > > > > > I would like to try this on my arm64 board. > > > > Glad to hear you are interested in this topic. > > > > > > According to your comments in the patch, I made this config file. > > # cat debian.conf > > name = "debian" > > type = "pvh" > > vcpus = 8 > > memory = 512 > > kernel = "/opt/agl/vmlinuz-5.9.0-1-arm64" > > ramdisk = "/opt/agl/initrd.img-5.9.0-1-arm64" > > cmdline= "console=hvc0 earlyprintk=xen root=/dev/xvda1 rw" > > disk = [ '/opt/agl/debian.qcow2,qcow2,hda' ] > > vif = [ 'mac=00:16:3E:74:3d:76,bridge=xenbr0' ] > > virtio = 1 > > vdisk = [ 'backend=Dom0, disks=ro:/dev/sda1' ] > > > > And tried to boot a DomU, but I got below error. > > > > # xl create -c debian.conf > > Parsing config from debian.conf > > libxl: error: libxl_create.c:1863:domcreate_attach_devices: Domain > > 1:unable to add virtio_disk devices > > libxl: error: libxl_domain.c:1218:destroy_domid_pci_done: Domain > > 1:xc_domain_pause failed > > libxl: error: libxl_dom.c:39:libxl__domain_type: unable to get > domain > > type for domid=1 > > libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain > > 1:Unable to destroy guest > > libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain > > 1:Destruction of domain failed > > > > > > Could you tell me how can I test it? > > > > > > I assume it is due to the lack of the virtio-disk backend (which I > haven't shared yet as I focused on the IOREQ/DM support on Arm in the > > first place). > > Could you wait a little bit, I am going to share it soon. > > Do you have a quick-and-dirty hack you can share in the meantime? Even > just on github as a special branch? It would be very useful to be able > to have a test-driver for the new feature. Well, I will provide a branch on github with our PoC virtio-disk backend by the end of this week. It will be possible to test this series with it.
On Thu, 29 Oct 2020, Oleksandr Tyshchenko wrote: > Hi Stefano > > [sorry for the possible format issue] > > On Thu, Oct 29, 2020 at 9:53 PM Stefano Stabellini <sstabellini@kernel.org> wrote: > On Thu, 29 Oct 2020, Oleksandr Tyshchenko wrote: > > On Thu, Oct 29, 2020 at 9:42 AM Masami Hiramatsu <masami.hiramatsu@linaro.org> wrote: > > Hi Oleksandr, > > > > Hi Masami > > > > [sorry for the possible format issue] > > > > > > I would like to try this on my arm64 board. > > > > Glad to hear you are interested in this topic. > > > > > > According to your comments in the patch, I made this config file. > > # cat debian.conf > > name = "debian" > > type = "pvh" > > vcpus = 8 > > memory = 512 > > kernel = "/opt/agl/vmlinuz-5.9.0-1-arm64" > > ramdisk = "/opt/agl/initrd.img-5.9.0-1-arm64" > > cmdline= "console=hvc0 earlyprintk=xen root=/dev/xvda1 rw" > > disk = [ '/opt/agl/debian.qcow2,qcow2,hda' ] > > vif = [ 'mac=00:16:3E:74:3d:76,bridge=xenbr0' ] > > virtio = 1 > > vdisk = [ 'backend=Dom0, disks=ro:/dev/sda1' ] > > > > And tried to boot a DomU, but I got below error. > > > > # xl create -c debian.conf > > Parsing config from debian.conf > > libxl: error: libxl_create.c:1863:domcreate_attach_devices: Domain > > 1:unable to add virtio_disk devices > > libxl: error: libxl_domain.c:1218:destroy_domid_pci_done: Domain > > 1:xc_domain_pause failed > > libxl: error: libxl_dom.c:39:libxl__domain_type: unable to get domain > > type for domid=1 > > libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain > > 1:Unable to destroy guest > > libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain > > 1:Destruction of domain failed > > > > > > Could you tell me how can I test it? > > > > > > I assume it is due to the lack of the virtio-disk backend (which I haven't shared yet as I focused on the IOREQ/DM support on > Arm in the > > first place). > > Could you wait a little bit, I am going to share it soon. > > Do you have a quick-and-dirty hack you can share in the meantime? Even > just on github as a special branch? It would be very useful to be able > to have a test-driver for the new feature. > > Well, I will provide a branch on github with our PoC virtio-disk backend by the end of this week. It will be possible to test this series > with it. Very good, thank you!
Hi Alex. [sorry for the possible format issues] > I assume this is wiring up the required bits of support to handle the > IOREQ requests in QEMU? No, it is not related to QEMU. We don't run QEMU in our system. Being honest I have never tried to run it) virtio-disk backend PoC is a completely standalone entity (IOREQ server) which emulates a virtio-mmio disk device. It is based on code from DEMU (for IOREQ server purposes) and some code from kvmtool to implement virtio protocol, disk operations over underlying H/W and Xenbus code to be able to read configuration from the Xenstore (it is configured via domain config file). Last patch in this series (marked as RFC) actually adds required bits to the libxl code. > We are putting together a PoC demo to show > a virtio enabled image (AGL) running on both KVM and Xen hypervisors so > we are keen to see your code as soon as you can share it. Thank you. Sure, I will provide a branch with code by the end of this week. > > I'm currently preparing a patch series for QEMU which fixes the recent > breakage caused by changes to the build system. As part of that I've > separated CONFIG_XEN and CONFIG_XEN_HVM so it's possible to build > i386-softmmu with unneeded for ARM bits. Hopefully this will allow me to > create a qemu-aarch64-system binary with just the PV related models in > it. > Does it mean that it will be possible to use QEMU in Xen on Arm just for "backend provider" purposes? > > Talking to Stefano it probably makes sense going forward to introduce a > new IOREQ aware machine type for QEMU that doesn't bring in the rest of > the x86 overhead. I was thinking maybe xenvirt? > > You've tested with virtio-block but we'd also like to extend this to > other arbitrary virtio devices. I guess we will need some sort of > mechanism to inform the QEMU command line where each device sits in the > virtio-mmio bus so the FDT Xen delivers to the guest matches up with > what QEMU is ready to serve requests for? > I am sorry, I can't provide ideas here, not familiar with QEMU. But, completely agree, that other virtio devices should be supported.
Hi Oleksandr, 2020年10月30日(金) 6:14 Oleksandr Tyshchenko <olekstysh@gmail.com>: > > Hi Stefano > > [sorry for the possible format issue] > > On Thu, Oct 29, 2020 at 9:53 PM Stefano Stabellini <sstabellini@kernel.org> wrote: >> >> On Thu, 29 Oct 2020, Oleksandr Tyshchenko wrote: >> > On Thu, Oct 29, 2020 at 9:42 AM Masami Hiramatsu <masami.hiramatsu@linaro.org> wrote: >> > Hi Oleksandr, >> > >> > Hi Masami >> > >> > [sorry for the possible format issue] >> > >> > >> > I would like to try this on my arm64 board. >> > >> > Glad to hear you are interested in this topic. >> > >> > >> > According to your comments in the patch, I made this config file. >> > # cat debian.conf >> > name = "debian" >> > type = "pvh" >> > vcpus = 8 >> > memory = 512 >> > kernel = "/opt/agl/vmlinuz-5.9.0-1-arm64" >> > ramdisk = "/opt/agl/initrd.img-5.9.0-1-arm64" >> > cmdline= "console=hvc0 earlyprintk=xen root=/dev/xvda1 rw" >> > disk = [ '/opt/agl/debian.qcow2,qcow2,hda' ] >> > vif = [ 'mac=00:16:3E:74:3d:76,bridge=xenbr0' ] >> > virtio = 1 >> > vdisk = [ 'backend=Dom0, disks=ro:/dev/sda1' ] >> > >> > And tried to boot a DomU, but I got below error. >> > >> > # xl create -c debian.conf >> > Parsing config from debian.conf >> > libxl: error: libxl_create.c:1863:domcreate_attach_devices: Domain >> > 1:unable to add virtio_disk devices >> > libxl: error: libxl_domain.c:1218:destroy_domid_pci_done: Domain >> > 1:xc_domain_pause failed >> > libxl: error: libxl_dom.c:39:libxl__domain_type: unable to get domain >> > type for domid=1 >> > libxl: error: libxl_domain.c:1136:domain_destroy_callback: Domain >> > 1:Unable to destroy guest >> > libxl: error: libxl_domain.c:1063:domain_destroy_cb: Domain >> > 1:Destruction of domain failed >> > >> > >> > Could you tell me how can I test it? >> > >> > >> > I assume it is due to the lack of the virtio-disk backend (which I haven't shared yet as I focused on the IOREQ/DM support on Arm in the >> > first place). >> > Could you wait a little bit, I am going to share it soon. >> >> Do you have a quick-and-dirty hack you can share in the meantime? Even >> just on github as a special branch? It would be very useful to be able >> to have a test-driver for the new feature. > > Well, I will provide a branch on github with our PoC virtio-disk backend by the end of this week. It will be possible to test this series with it. Great! OK I'll be waiting for the PoC backend. Thank you!
On Fri, Oct 30, 2020 at 1:34 PM Masami Hiramatsu < masami.hiramatsu@linaro.org> wrote: > Hi Oleksandr, > Hi Masami, all [sorry for the possible format issue] > >> > > >> > Could you tell me how can I test it? > >> > > >> > > >> > I assume it is due to the lack of the virtio-disk backend (which I > haven't shared yet as I focused on the IOREQ/DM support on Arm in the > >> > first place). > >> > Could you wait a little bit, I am going to share it soon. > >> > >> Do you have a quick-and-dirty hack you can share in the meantime? Even > >> just on github as a special branch? It would be very useful to be able > >> to have a test-driver for the new feature. > > > > Well, I will provide a branch on github with our PoC virtio-disk backend > by the end of this week. It will be possible to test this series with it. > > Great! OK I'll be waiting for the PoC backend. > > Thank you! > You can find the virtio-disk backend PoC (shared as is) at [1]. Brief description... The virtio-disk backend PoC is a completely standalone entity (IOREQ server) which emulates a virtio-mmio disk device. It is based on code from DEMU [2] (for IOREQ server purposes) and some code from kvmtool [3] to implement virtio protocol, disk operations over underlying H/W and Xenbus code to be able to read configuration from the Xenstore (it is configured via domain config file). Last patch in this series (marked as RFC) actually adds required bits to the libxl code. Some notes... Backend could be used with current V2 IOREQ series [4] without any modifications, all what you need is to enable CONFIG_IOREQ_SERVER on Arm [5], since it is disabled by default within this series. Please note that in our system we run backend in DomD (driver domain). I haven't tested it in Dom0, since in our system the Dom0 is thin (without any H/W) and only used to launch VMs, so there is no underlying block H/W. But, I hope, it is possible to run it in Dom0 as well (at least there is nothing specific to a particular domain in the backend itself, nothing hardcoded). If you are going to run a backend in other than Dom0 domain you need to write your own policy (FLASK) for the backend (running in that domain) to be able to issue DM related requests, etc. Only for test purposes you could use this patch [6] that tweeks Xen dummy policy (not for upstream). As I mentioned elsewhere you don't need to modify Guest Linux (DomU), just enable VirtIO related configs. If I remember correctly, the following would be enough: CONFIG_BLK_MQ_VIRTIO=y CONFIG_VIRTIO_BLK=y CONFIG_VIRTIO=y CONFIG_VIRTIO_BALLOON=y CONFIG_VIRTIO_MMIO=y If I remember correctly, if your Host Linux (Dom0 or DomD) version >= 4.17 you don't need to modify it as well. Otherwise, you need to cherry-pick "xen/privcmd: add IOCTL_PRIVCMD_MMAP_RESOURCE" from the upstream to be able to use the acquire interface for the resource mapping. We usually build a backend in the context of the Yocto build process and run it as a systemd service, but you can also build and run it manually (it should be launched before DomU creation). There are no command line options at all. Everything is configured via domain configuration file: # This option is mandatory, it shows that VirtIO is going to be used by guest virtio=1 # Example of domain configuration (two disks are assigned to the guest, the latter is in readonly mode): vdisk = [ 'backend=DomD, disks=rw:/dev/mmcblk0p3;ro:/dev/mmcblk1p3' ] Hope that helps. Feel free to ask questions if any. [1] https://github.com/xen-troops/virtio-disk/commits/ioreq_v3 [2] https://xenbits.xen.org/gitweb/?p=people/pauldu/demu.git;a=summary [3] https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git/ [4] https://github.com/otyshchenko1/xen/commits/ioreq_4.14_ml3 [5] https://github.com/otyshchenko1/xen/commit/ee221102193f0422a240832edc41d73f6f3da923 [6] https://github.com/otyshchenko1/xen/commit/be868a63014b7aa6c9731d5692200d7f2f57c611
Hi Oleksandr,
Thanks for the sharing of virtio-disk backend. I have tested it on arm FVP_base platform.
We used domain-0 to run virtio disk backend. The backend disk is a loop device.
"virtio_disks": [
{
"backend_domname": "Domain-0",
"devid": 0,
"disks": [
{
"filename": "/dev/loop0"
}
]
}
],
It works fine and I've pasted some logs:
-------------------------------------------
Domain-0 logs:
main: read backend domid 0
(XEN) gnttab_mark_dirty not implemented yet
(XEN) domain_direct_pl011_init for domain#2
main: read frontend domid 2
Info: connected to dom2
demu_seq_next: >XENSTORE_ATTACHED
demu_seq_next: domid = 2
demu_seq_next: filename[0] = /dev/loop0
demu_seq_next: readonly[0] = 0
demu_seq_next: base[0] = 0x2000000
demu_seq_next: irq[0] = 33
demu_seq_next: >XENCTRL_OPEN
demu_seq_next: >XENEVTCHN_OPEN
demu_seq_next: >XENFOREIGNMEMORY_OPEN
demu_seq_next: >XENDEVICEMODEL_OPEN
demu_initialize: 2 vCPU(s)
demu_seq_next: >SERVER_REGISTERED
demu_seq_next: ioservid = 0
demu_seq_next: >RESOURCE_MAPPED
demu_seq_next: shared_iopage = 0xffffae6de000
demu_seq_next: buffered_iopage = 0xffffae6dd000
demu_seq_next: >SERVER_ENABLED
demu_seq_next: >PORT_ARRAY_ALLOCATED
demu_seq_next: >EVTCHN_PORTS_BOUND
demu_seq_next: VCPU0: 3 -> 7
demu_seq_next: VCPU1: 5 -> 8
demu_seq_next: >EVTCHN_BUF_PORT_BOUND
demu_seq_next: 0 -> 9
demu_register_memory_space: 2000000 - 20001ff
Info: (virtio/mmio.c) virtio_mmio_init:290: mailto:virtio-mmio.devices=0x200@0x2000000:33
demu_seq_next: >DEVICE_INITIALIZED
demu_seq_next: >INITIALIZED
IO request not ready
IO request not ready
----------------
Dom-U logs:
[ 0.491037] xen:xen_evtchn: Event-channel device installed
[ 0.493600] Initialising Xen pvcalls frontend driver
[ 0.516807] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[ 0.525565] cacheinfo: Unable to detect cache hierarchy for CPU 0
[ 0.562275] brd: module loaded
[ 0.595300] loop: module loaded
[ 0.683800] virtio_blk virtio0: [vda] 131072 512-byte logical blocks (67.1 MB/64.0 MiB)
[ 0.684000] vda: detected capacity change from 0 to 67108864
/ # dd if=/dev/vda of=/dev/null bs=1M count=64
64+0 records in
64+0 records out
67108864 bytes (64.0MB) copied, 3.196242 seconds, 20.0MB/s
/ # dd if=/dev/zero of=/dev/vda bs=1M count=64
64+0 records in
64+0 records out
67108864 bytes (64.0MB) copied, 3.704594 seconds, 17.3MB/s
---------------------
The read/write seems OK in dom-U. The FVP platform is a emulator, so the performance is no reference.
We will test it on real hardware like N1SDP.
Thanks,
Wei Chen
----------------------------------------------------------------------------------
From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Oleksandr Tyshchenko
Sent: 2020年11月1日 5:11
To: Masami Hiramatsu <masami.hiramatsu@linaro.org>; Alex Bennée <alex.bennee@linaro.org>
Cc: Stefano Stabellini <sstabellini@kernel.org>; xen-devel <xen-devel@lists.xenproject.org>; Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>; Paul Durrant <paul@xen.org>; Jan Beulich <jbeulich@suse.com>; Andrew Cooper <andrew.cooper3@citrix.com>; Roger Pau Monné <roger.pau@citrix.com>; Wei Liu <wl@xen.org>; Julien Grall <Julien.Grall@arm.com>; George Dunlap <george.dunlap@citrix.com>; Ian Jackson <iwj@xenproject.org>; Julien Grall <julien@xen.org>; Tim Deegan <tim@xen.org>; Daniel De Graaf <dgdegra@tycho.nsa.gov>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Jun Nakajima <jun.nakajima@intel.com>; Kevin Tian <kevin.tian@intel.com>; Anthony PERARD <anthony.perard@citrix.com>; Bertrand Marquis <Bertrand.Marquis@arm.com>
Subject: Re: [PATCH V2 00/23] IOREQ feature (+ virtio-mmio) on Arm
On Fri, Oct 30, 2020 at 1:34 PM Masami Hiramatsu <mailto:masami.hiramatsu@linaro.org> wrote:
Hi Oleksandr,
Hi Masami, all
[sorry for the possible format issue]
>> >
>> > Could you tell me how can I test it?
>> >
>> >
>> > I assume it is due to the lack of the virtio-disk backend (which I haven't shared yet as I focused on the IOREQ/DM support on Arm in the
>> > first place).
>> > Could you wait a little bit, I am going to share it soon.
>>
>> Do you have a quick-and-dirty hack you can share in the meantime? Even
>> just on github as a special branch? It would be very useful to be able
>> to have a test-driver for the new feature.
>
> Well, I will provide a branch on github with our PoC virtio-disk backend by the end of this week. It will be possible to test this series with it.
Great! OK I'll be waiting for the PoC backend.
Thank you!
You can find the virtio-disk backend PoC (shared as is) at [1].
Brief description...
The virtio-disk backend PoC is a completely standalone entity (IOREQ server) which emulates a virtio-mmio disk device.
It is based on code from DEMU [2] (for IOREQ server purposes) and some code from kvmtool [3] to implement virtio protocol,
disk operations over underlying H/W and Xenbus code to be able to read configuration from the Xenstore
(it is configured via domain config file). Last patch in this series (marked as RFC) actually adds required bits to the libxl code.
Some notes...
Backend could be used with current V2 IOREQ series [4] without any modifications, all what you need is to enable
CONFIG_IOREQ_SERVER on Arm [5], since it is disabled by default within this series.
Please note that in our system we run backend in DomD (driver domain). I haven't tested it in Dom0,
since in our system the Dom0 is thin (without any H/W) and only used to launch VMs, so there is no underlying block H/W.
But, I hope, it is possible to run it in Dom0 as well (at least there is nothing specific to a particular domain in the backend itself, nothing hardcoded).
If you are going to run a backend in other than Dom0 domain you need to write your own policy (FLASK) for the backend (running in that domain)
to be able to issue DM related requests, etc. Only for test purposes you could use this patch [6] that tweeks Xen dummy policy (not for upstream).
As I mentioned elsewhere you don't need to modify Guest Linux (DomU), just enable VirtIO related configs.
If I remember correctly, the following would be enough:
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_MMIO=y
If I remember correctly, if your Host Linux (Dom0 or DomD) version >= 4.17 you don't need to modify it as well.
Otherwise, you need to cherry-pick "xen/privcmd: add IOCTL_PRIVCMD_MMAP_RESOURCE" from the upstream to be able
to use the acquire interface for the resource mapping.
We usually build a backend in the context of the Yocto build process and run it as a systemd service,
but you can also build and run it manually (it should be launched before DomU creation).
There are no command line options at all. Everything is configured via domain configuration file:
# This option is mandatory, it shows that VirtIO is going to be used by guest
virtio=1
# Example of domain configuration (two disks are assigned to the guest, the latter is in readonly mode):
vdisk = [ 'backend=DomD, disks=rw:/dev/mmcblk0p3;ro:/dev/mmcblk1p3' ]
Hope that helps. Feel free to ask questions if any.
[1] https://github.com/xen-troops/virtio-disk/commits/ioreq_v3
[2] https://xenbits.xen.org/gitweb/?p=people/pauldu/demu.git;a=summary
[3] https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git/
[4] https://github.com/otyshchenko1/xen/commits/ioreq_4.14_ml3
[5] https://github.com/otyshchenko1/xen/commit/ee221102193f0422a240832edc41d73f6f3da923
[6] https://github.com/otyshchenko1/xen/commit/be868a63014b7aa6c9731d5692200d7f2f57c611
On 02.11.20 09:23, Wei Chen wrote: > Hi Oleksandr, Hi Wei. > > Thanks for the sharing of virtio-disk backend. I have tested it on arm FVP_base platform. > We used domain-0 to run virtio disk backend. The backend disk is a loop device. > "virtio_disks": [ > { > "backend_domname": "Domain-0", > "devid": 0, > "disks": [ > { > "filename": "/dev/loop0" > } > ] > } > ], > > It works fine and I've pasted some logs: > > ------------------------------------------- > Domain-0 logs: > main: read backend domid 0 > (XEN) gnttab_mark_dirty not implemented yet > (XEN) domain_direct_pl011_init for domain#2 > main: read frontend domid 2 > Info: connected to dom2 > > demu_seq_next: >XENSTORE_ATTACHED > demu_seq_next: domid = 2 > demu_seq_next: filename[0] = /dev/loop0 > demu_seq_next: readonly[0] = 0 > demu_seq_next: base[0] = 0x2000000 > demu_seq_next: irq[0] = 33 > demu_seq_next: >XENCTRL_OPEN > demu_seq_next: >XENEVTCHN_OPEN > demu_seq_next: >XENFOREIGNMEMORY_OPEN > demu_seq_next: >XENDEVICEMODEL_OPEN > demu_initialize: 2 vCPU(s) > demu_seq_next: >SERVER_REGISTERED > demu_seq_next: ioservid = 0 > demu_seq_next: >RESOURCE_MAPPED > demu_seq_next: shared_iopage = 0xffffae6de000 > demu_seq_next: buffered_iopage = 0xffffae6dd000 > demu_seq_next: >SERVER_ENABLED > demu_seq_next: >PORT_ARRAY_ALLOCATED > demu_seq_next: >EVTCHN_PORTS_BOUND > demu_seq_next: VCPU0: 3 -> 7 > demu_seq_next: VCPU1: 5 -> 8 > demu_seq_next: >EVTCHN_BUF_PORT_BOUND > demu_seq_next: 0 -> 9 > demu_register_memory_space: 2000000 - 20001ff > Info: (virtio/mmio.c) virtio_mmio_init:290: mailto:virtio-mmio.devices=0x200@0x2000000:33 > demu_seq_next: >DEVICE_INITIALIZED > demu_seq_next: >INITIALIZED > IO request not ready > IO request not ready > > ---------------- > Dom-U logs: > [ 0.491037] xen:xen_evtchn: Event-channel device installed > [ 0.493600] Initialising Xen pvcalls frontend driver > [ 0.516807] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled > [ 0.525565] cacheinfo: Unable to detect cache hierarchy for CPU 0 > [ 0.562275] brd: module loaded > [ 0.595300] loop: module loaded > [ 0.683800] virtio_blk virtio0: [vda] 131072 512-byte logical blocks (67.1 MB/64.0 MiB) > [ 0.684000] vda: detected capacity change from 0 to 67108864 > > > / # dd if=/dev/vda of=/dev/null bs=1M count=64 > 64+0 records in > 64+0 records out > 67108864 bytes (64.0MB) copied, 3.196242 seconds, 20.0MB/s > / # dd if=/dev/zero of=/dev/vda bs=1M count=64 > 64+0 records in > 64+0 records out > 67108864 bytes (64.0MB) copied, 3.704594 seconds, 17.3MB/s > --------------------- > > The read/write seems OK in dom-U. The FVP platform is a emulator, so the performance is no reference. > We will test it on real hardware like N1SDP. This is really a good news. Thank you for testing!
Hi Oleksandr, Thanks for sharing the virtio-disk server, I also tested with a real usb disk. In config file: virtio = 1 vdisk = [ 'backend=Domain-0, disks=ro:/dev/sda1' ] And it can be mounted in DomU [ 2.892874] virtio_blk virtio0: [vda] 1875382927 512-byte logical blocks (960 GB/894 GiB) [ 2.892925] vda: detected capacity change from 0 to 960196058624 ... root@develbox:~# cat /proc/partitions major minor #blocks name 254 0 937691463 vda ... root@develbox:~# mount /dev/vda /mnt/ [ 192.260968] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null) mount: /mnt: WARNING: source write-protected, mounted read-only. So "ro" flag also correctly works. Great! Thank you! 2020年11月1日(日) 6:10 Oleksandr Tyshchenko <olekstysh@gmail.com>: > > > > On Fri, Oct 30, 2020 at 1:34 PM Masami Hiramatsu <masami.hiramatsu@linaro.org> wrote: >> >> Hi Oleksandr, > > > Hi Masami, all > > [sorry for the possible format issue] > >> >> >> > >> >> > Could you tell me how can I test it? >> >> > >> >> > >> >> > I assume it is due to the lack of the virtio-disk backend (which I haven't shared yet as I focused on the IOREQ/DM support on Arm in the >> >> > first place). >> >> > Could you wait a little bit, I am going to share it soon. >> >> >> >> Do you have a quick-and-dirty hack you can share in the meantime? Even >> >> just on github as a special branch? It would be very useful to be able >> >> to have a test-driver for the new feature. >> > >> > Well, I will provide a branch on github with our PoC virtio-disk backend by the end of this week. It will be possible to test this series with it. >> >> Great! OK I'll be waiting for the PoC backend. >> >> Thank you! > > > You can find the virtio-disk backend PoC (shared as is) at [1]. > > Brief description... > > The virtio-disk backend PoC is a completely standalone entity (IOREQ server) which emulates a virtio-mmio disk device. > It is based on code from DEMU [2] (for IOREQ server purposes) and some code from kvmtool [3] to implement virtio protocol, > disk operations over underlying H/W and Xenbus code to be able to read configuration from the Xenstore > (it is configured via domain config file). Last patch in this series (marked as RFC) actually adds required bits to the libxl code. > > Some notes... > > Backend could be used with current V2 IOREQ series [4] without any modifications, all what you need is to enable > CONFIG_IOREQ_SERVER on Arm [5], since it is disabled by default within this series. > > Please note that in our system we run backend in DomD (driver domain). I haven't tested it in Dom0, > since in our system the Dom0 is thin (without any H/W) and only used to launch VMs, so there is no underlying block H/W. > But, I hope, it is possible to run it in Dom0 as well (at least there is nothing specific to a particular domain in the backend itself, nothing hardcoded). > If you are going to run a backend in other than Dom0 domain you need to write your own policy (FLASK) for the backend (running in that domain) > to be able to issue DM related requests, etc. Only for test purposes you could use this patch [6] that tweeks Xen dummy policy (not for upstream). > > As I mentioned elsewhere you don't need to modify Guest Linux (DomU), just enable VirtIO related configs. > If I remember correctly, the following would be enough: > CONFIG_BLK_MQ_VIRTIO=y > CONFIG_VIRTIO_BLK=y > CONFIG_VIRTIO=y > CONFIG_VIRTIO_BALLOON=y > CONFIG_VIRTIO_MMIO=y > If I remember correctly, if your Host Linux (Dom0 or DomD) version >= 4.17 you don't need to modify it as well. > Otherwise, you need to cherry-pick "xen/privcmd: add IOCTL_PRIVCMD_MMAP_RESOURCE" from the upstream to be able > to use the acquire interface for the resource mapping. > > We usually build a backend in the context of the Yocto build process and run it as a systemd service, > but you can also build and run it manually (it should be launched before DomU creation). > > There are no command line options at all. Everything is configured via domain configuration file: > # This option is mandatory, it shows that VirtIO is going to be used by guest > virtio=1 > # Example of domain configuration (two disks are assigned to the guest, the latter is in readonly mode): > vdisk = [ 'backend=DomD, disks=rw:/dev/mmcblk0p3;ro:/dev/mmcblk1p3' ] > > Hope that helps. Feel free to ask questions if any. > > [1] https://github.com/xen-troops/virtio-disk/commits/ioreq_v3 > [2] https://xenbits.xen.org/gitweb/?p=people/pauldu/demu.git;a=summary > [3] https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git/ > [4] https://github.com/otyshchenko1/xen/commits/ioreq_4.14_ml3 > [5] https://github.com/otyshchenko1/xen/commit/ee221102193f0422a240832edc41d73f6f3da923 > [6] https://github.com/otyshchenko1/xen/commit/be868a63014b7aa6c9731d5692200d7f2f57c611 > > -- > Regards, > > Oleksandr Tyshchenko
On 03.11.20 16:30, Masami Hiramatsu wrote: > Hi Oleksandr, Hi Masami > > Thanks for sharing the virtio-disk server, I also tested with a real usb disk. > > In config file: > > virtio = 1 > vdisk = [ 'backend=Domain-0, disks=ro:/dev/sda1' ] > > And it can be mounted in DomU > > [ 2.892874] virtio_blk virtio0: [vda] 1875382927 512-byte logical > blocks (960 GB/894 GiB) > [ 2.892925] vda: detected capacity change from 0 to 960196058624 > ... > root@develbox:~# cat /proc/partitions > major minor #blocks name > > 254 0 937691463 vda > ... > root@develbox:~# mount /dev/vda /mnt/ > [ 192.260968] EXT4-fs (vda): mounted filesystem with ordered data > mode. Opts: (null) > mount: /mnt: WARNING: source write-protected, mounted read-only. > > So "ro" flag also correctly works. > Great! > > Thank you! Sounds great. Thank you for testing!
From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> Hello all. The purpose of this patch series is to add IOREQ/DM support to Xen on Arm. You can find an initial discussion at [1] and RFC/V1 series at [2]/[3]. Xen on Arm requires some implementation to forward guest MMIO access to a device model in order to implement virtio-mmio backend or even mediator outside of hypervisor. As Xen on x86 already contains required support this series tries to make it common and introduce Arm specific bits plus some new functionality. Patch series is based on Julien's PoC "xen/arm: Add support for Guest IO forwarding to a device emulator". Besides splitting existing IOREQ/DM support and introducing Arm side, the series also includes virtio-mmio related changes (last 2 patches for toolstack) for the reviewers to be able to see how the whole picture could look like. According to the initial discussion there are a few open questions/concerns regarding security, performance in VirtIO solution: 1. virtio-mmio vs virtio-pci, SPI vs MSI, different use-cases require different transport... 2. virtio backend is able to access all guest memory, some kind of protection is needed: 'virtio-iommu in Xen' vs 'pre-shared-memory & memcpys in guest' 3. interface between toolstack and 'out-of-qemu' virtio backend, avoid using Xenstore in virtio backend if possible. 4. a lot of 'foreing mapping' could lead to the memory exhaustion, Julien has some idea regarding that. Looks like all of them are valid and worth considering, but the first thing which we need on Arm is a mechanism to forward guest IO to a device emulator, so let's focus on it in the first place. *** There are a lot of changes since RFC series, almost all TODOs were resolved on Arm, Arm code was improved and hardened, common IOREQ/DM code became really arch-agnostic (without HVM-ism), but one TODO still remains which is "PIO handling" on Arm. The "PIO handling" TODO is expected to left unaddressed for the current series. It is not an big issue for now while Xen doesn't have support for vPCI on Arm. On Arm64 they are only used for PCI IO Bar and we would probably want to expose them to emulator as PIO access to make a DM completely arch-agnostic. So "PIO handling" should be implemented when we add support for vPCI. I left interface untouched in the following patch "xen/dm: Introduce xendevicemodel_set_irq_level DM op" since there is still an open discussion what interface to use/what information to pass to the hypervisor. Also I decided to drop the following patch: "[RFC PATCH V1 07/12] A collection of tweaks to be able to run emulator in driver domain" as I got an advise to write our own policy using FLASK which would cover our use case (with emulator in driver domain) rather than tweak Xen. There are two patches on review this series depends on (each involved patch in this series contains this note as well): 1. https://patchwork.kernel.org/patch/11816689 2. https://patchwork.kernel.org/patch/11803383 Please note, that IOREQ feature is disabled by default within this series. *** Patch series [4] was rebased on recent "staging branch" (8a62dee x86/vLAPIC: don't leak regs page from vlapic_init() upon error) and tested on Renesas Salvator-X board + H3 ES3.0 SoC (Arm64) with virtio-mmio disk backend (we will share it later) running in driver domain and unmodified Linux Guest running on existing virtio-blk driver (frontend). No issues were observed. Guest domain 'reboot/destroy' use-cases work properly. Patch series was only build-tested on x86. Please note, build-test passed for the following modes: 1. x86: CONFIG_HVM=y / CONFIG_IOREQ_SERVER=y (default) 2. x86: #CONFIG_HVM is not set / #CONFIG_IOREQ_SERVER is not set 3. Arm64: CONFIG_HVM=y / CONFIG_IOREQ_SERVER=y 4. Arm64: CONFIG_HVM=y / #CONFIG_IOREQ_SERVER is not set (default) 5. Arm32: CONFIG_HVM=y / CONFIG_IOREQ_SERVER=y 6. Arm32: CONFIG_HVM=y / #CONFIG_IOREQ_SERVER is not set (default) *** Any feedback/help would be highly appreciated. [1] https://lists.xenproject.org/archives/html/xen-devel/2020-07/msg00825.html [2] https://lists.xenproject.org/archives/html/xen-devel/2020-08/msg00071.html [3] https://lists.xenproject.org/archives/html/xen-devel/2020-09/msg00732.html [4] https://github.com/otyshchenko1/xen/commits/ioreq_4.14_ml3 Julien Grall (5): xen/dm: Make x86's DM feature common xen/mm: Make x86's XENMEM_resource_ioreq_server handling common arm/ioreq: Introduce arch specific bits for IOREQ/DM features xen/dm: Introduce xendevicemodel_set_irq_level DM op libxl: Introduce basic virtio-mmio support on Arm Oleksandr Tyshchenko (18): x86/ioreq: Prepare IOREQ feature for making it common xen/ioreq: Make x86's IOREQ feature common xen/ioreq: Make x86's hvm_ioreq_needs_completion() common xen/ioreq: Provide alias for the handle_mmio() xen/ioreq: Make x86's hvm_mmio_first(last)_byte() common xen/ioreq: Make x86's hvm_ioreq_(page/vcpu/server) structs common xen/ioreq: Move x86's ioreq_gfn(server) to struct domain xen/ioreq: Introduce ioreq_params to abstract accesses to arch.hvm.params xen/ioreq: Move x86's io_completion/io_req fields to struct vcpu xen/ioreq: Remove "hvm" prefixes from involved function names xen/ioreq: Use guest_cmpxchg64() instead of cmpxchg() xen/arm: Stick around in leave_hypervisor_to_guest until I/O has completed xen/mm: Handle properly reference in set_foreign_p2m_entry() on Arm xen/ioreq: Introduce domain_has_ioreq_server() xen/arm: io: Abstract sign-extension xen/ioreq: Make x86's send_invalidate_req() common xen/arm: Add mapcache invalidation handling [RFC] libxl: Add support for virtio-disk configuration MAINTAINERS | 8 +- tools/libs/devicemodel/core.c | 18 + tools/libs/devicemodel/include/xendevicemodel.h | 4 + tools/libs/devicemodel/libxendevicemodel.map | 1 + tools/libs/light/Makefile | 1 + tools/libs/light/libxl_arm.c | 94 +- tools/libs/light/libxl_create.c | 1 + tools/libs/light/libxl_internal.h | 1 + tools/libs/light/libxl_types.idl | 16 + tools/libs/light/libxl_types_internal.idl | 1 + tools/libs/light/libxl_virtio_disk.c | 109 ++ tools/xl/Makefile | 2 +- tools/xl/xl.h | 3 + tools/xl/xl_cmdtable.c | 15 + tools/xl/xl_parse.c | 116 ++ tools/xl/xl_virtio_disk.c | 46 + xen/arch/arm/Makefile | 2 + xen/arch/arm/dm.c | 89 ++ xen/arch/arm/domain.c | 9 + xen/arch/arm/hvm.c | 4 + xen/arch/arm/io.c | 29 +- xen/arch/arm/ioreq.c | 126 ++ xen/arch/arm/p2m.c | 29 + xen/arch/arm/traps.c | 58 +- xen/arch/x86/Kconfig | 1 + xen/arch/x86/hvm/Makefile | 1 - xen/arch/x86/hvm/dm.c | 291 +---- xen/arch/x86/hvm/emulate.c | 60 +- xen/arch/x86/hvm/hvm.c | 24 +- xen/arch/x86/hvm/hypercall.c | 9 +- xen/arch/x86/hvm/intercept.c | 5 +- xen/arch/x86/hvm/io.c | 26 +- xen/arch/x86/hvm/ioreq.c | 1533 ----------------------- xen/arch/x86/hvm/stdvga.c | 10 +- xen/arch/x86/hvm/svm/nestedsvm.c | 2 +- xen/arch/x86/hvm/vmx/realmode.c | 6 +- xen/arch/x86/hvm/vmx/vvmx.c | 2 +- xen/arch/x86/mm.c | 46 +- xen/arch/x86/mm/p2m.c | 13 +- xen/arch/x86/mm/shadow/common.c | 2 +- xen/common/Kconfig | 3 + xen/common/Makefile | 2 + xen/common/dm.c | 292 +++++ xen/common/ioreq.c | 1443 +++++++++++++++++++++ xen/common/memory.c | 50 +- xen/include/asm-arm/domain.h | 5 + xen/include/asm-arm/hvm/ioreq.h | 109 ++ xen/include/asm-arm/mm.h | 8 - xen/include/asm-arm/mmio.h | 1 + xen/include/asm-arm/p2m.h | 19 +- xen/include/asm-arm/paging.h | 4 + xen/include/asm-arm/traps.h | 24 + xen/include/asm-x86/hvm/domain.h | 50 +- xen/include/asm-x86/hvm/emulate.h | 2 +- xen/include/asm-x86/hvm/io.h | 17 - xen/include/asm-x86/hvm/ioreq.h | 198 ++- xen/include/asm-x86/hvm/vcpu.h | 18 - xen/include/asm-x86/mm.h | 4 - xen/include/asm-x86/p2m.h | 20 +- xen/include/public/arch-arm.h | 5 + xen/include/public/hvm/dm_op.h | 16 + xen/include/xen/dm.h | 44 + xen/include/xen/ioreq.h | 143 +++ xen/include/xen/p2m-common.h | 4 + xen/include/xen/sched.h | 37 + xen/include/xsm/dummy.h | 4 +- xen/include/xsm/xsm.h | 6 +- xen/xsm/dummy.c | 2 +- xen/xsm/flask/hooks.c | 5 +- 69 files changed, 3223 insertions(+), 2125 deletions(-) create mode 100644 tools/libs/light/libxl_virtio_disk.c create mode 100644 tools/xl/xl_virtio_disk.c create mode 100644 xen/arch/arm/dm.c create mode 100644 xen/arch/arm/ioreq.c delete mode 100644 xen/arch/x86/hvm/ioreq.c create mode 100644 xen/common/dm.c create mode 100644 xen/common/ioreq.c create mode 100644 xen/include/asm-arm/hvm/ioreq.h create mode 100644 xen/include/xen/dm.h create mode 100644 xen/include/xen/ioreq.h