[v7,9/9] docs/system: add basic virtio-gpu documentation

Commit Message

Gurchetan Singh Aug. 17, 2023, 2:23 a.m. UTC

From: Gurchetan Singh <[e-mail]>

This adds basic documentation for virtio-gpu.

Suggested-by: Akihiko Odaki <[e-mail]>
Signed-off-by: Gurchetan Singh <[e-mail]>
Tested-by: Alyssa Ross <[e-mail]>
Tested-by: Emmanouil Pitsidianakis <[e-mail]>
Reviewed-by: Emmanouil Pitsidianakis <[e-mail]>
---
v2: - Incorporated suggestions by Akihiko Odaki
    - Listed the currently supported capset_names (Bernard)

v3: - Incorporated suggestions by Akihiko Odaki and Alyssa Ross

v4: - Incorporated suggestions by Akihiko Odaki

v5: - Removed pci suffix from examples
    - Verified that -device virtio-gpu-rutabaga works.  Strangely
      enough, I don't remember changing anything, and I remember
      it not working.  I did rebase to top of tree though.
    - Fixed meson examples in crosvm docs

 docs/system/device-emulation.rst   |   1 +
 docs/system/devices/virtio-gpu.rst | 113 +++++++++++++++++++++++++++++
 2 files changed, 114 insertions(+)
 create mode 100644 docs/system/devices/virtio-gpu.rst

Comments

Akihiko Odaki Aug. 17, 2023, 5:28 a.m. UTC | #1

On 2023/08/17 11:23, Gurchetan Singh wrote:
> From: Gurchetan Singh <[e-mail]>
> 
> This adds basic documentation for virtio-gpu.
> 
> Suggested-by: Akihiko Odaki <[e-mail]>
> Signed-off-by: Gurchetan Singh <[e-mail]>
> Tested-by: Alyssa Ross <[e-mail]>
> Tested-by: Emmanouil Pitsidianakis <[e-mail]>
> Reviewed-by: Emmanouil Pitsidianakis <[e-mail]>
> ---
> v2: - Incorporated suggestions by Akihiko Odaki
>      - Listed the currently supported capset_names (Bernard)
> 
> v3: - Incorporated suggestions by Akihiko Odaki and Alyssa Ross
> 
> v4: - Incorporated suggestions by Akihiko Odaki
> 
> v5: - Removed pci suffix from examples
>      - Verified that -device virtio-gpu-rutabaga works.  Strangely
>        enough, I don't remember changing anything, and I remember
>        it not working.  I did rebase to top of tree though.
>      - Fixed meson examples in crosvm docs
> 
>   docs/system/device-emulation.rst   |   1 +
>   docs/system/devices/virtio-gpu.rst | 113 +++++++++++++++++++++++++++++
>   2 files changed, 114 insertions(+)
>   create mode 100644 docs/system/devices/virtio-gpu.rst
> 
> diff --git a/docs/system/device-emulation.rst b/docs/system/device-emulation.rst
> index 4491c4cbf7..1167f3a9f2 100644
> --- a/docs/system/device-emulation.rst
> +++ b/docs/system/device-emulation.rst
> @@ -91,6 +91,7 @@ Emulated Devices
>      devices/nvme.rst
>      devices/usb.rst
>      devices/vhost-user.rst
> +   devices/virtio-gpu.rst
>      devices/virtio-pmem.rst
>      devices/vhost-user-rng.rst
>      devices/canokey.rst
> diff --git a/docs/system/devices/virtio-gpu.rst b/docs/system/devices/virtio-gpu.rst
> new file mode 100644
> index 0000000000..8c5c708272
> --- /dev/null
> +++ b/docs/system/devices/virtio-gpu.rst
> @@ -0,0 +1,113 @@
> +..
> +   SPDX-License-Identifier: GPL-2.0
> +
> +virtio-gpu
> +==========
> +
> +This document explains the setup and usage of the virtio-gpu device.
> +The virtio-gpu device paravirtualizes the GPU and display controller.
> +
> +Linux kernel support
> +--------------------
> +
> +virtio-gpu requires a guest Linux kernel built with the
> +``CONFIG_DRM_VIRTIO_GPU`` option.
> +
> +QEMU virtio-gpu variants
> +------------------------
> +
> +QEMU virtio-gpu device variants come in the following form:
> +
> + * ``virtio-vga[-BACKEND]``
> + * ``virtio-gpu[-BACKEND][-INTERFACE]``
> + * ``vhost-user-vga``
> + * ``vhost-user-pci``
> +
> +**Backends:** QEMU provides a 2D virtio-gpu backend, and two accelerated
> +backends: virglrenderer ('gl' device label) and rutabaga_gfx ('rutabaga'
> +device label).  There is a vhost-user backend that runs the graphics stack
> +in a separate process for improved isolation.
> +
> +**Interfaces:** QEMU further categorizes virtio-gpu device variants based
> +on the interface exposed to the guest. The interfaces can be classified
> +into VGA and non-VGA variants. The VGA ones are prefixed with virtio-vga
> +or vhost-user-vga while the non-VGA ones are prefixed with virtio-gpu or
> +vhost-user-gpu.
> +
> +The VGA ones always use the PCI interface, but for the non-VGA ones, the
> +user can further pick between MMIO or PCI. For MMIO, the user can suffix
> +the device name with -device, though vhost-user-gpu does not support MMIO.
> +For PCI, the user can suffix it with -pci. Without these suffixes, the
> +platform default will be chosen.
> +
> +virtio-gpu 2d
> +-------------
> +
> +The default 2D backend only performs 2D operations. The guest needs to
> +employ a software renderer for 3D graphics.
> +
> +Typically, the software renderer is provided by `Mesa`_ or `SwiftShader`_.
> +Mesa's implementations (LLVMpipe, Lavapipe and virgl below) work out of box
> +on typical modern Linux distributions.
> +
> +.. parsed-literal::
> +    -device virtio-gpu
> +
> +.. _Mesa: https://www.mesa3d.org/
> +.. _SwiftShader: https://github.com/google/swiftshader
> +
> +virtio-gpu virglrenderer
> +------------------------
> +
> +When using virgl accelerated graphics mode in the guest, OpenGL API calls
> +are translated into an intermediate representation (see `Gallium3D`_). The
> +intermediate representation is communicated to the host and the
> +`virglrenderer`_ library on the host translates the intermediate
> +representation back to OpenGL API calls.
> +
> +.. parsed-literal::
> +    -device virtio-gpu-gl
> +
> +.. _Gallium3D: https://www.freedesktop.org/wiki/Software/gallium/
> +.. _virglrenderer: https://gitlab.freedesktop.org/virgl/virglrenderer/
> +
> +virtio-gpu rutabaga
> +-------------------
> +
> +virtio-gpu can also leverage `rutabaga_gfx`_ to provide `gfxstream`_
> +rendering and `Wayland display passthrough`_.  With the gfxstream rendering
> +mode, GLES and Vulkan calls are forwarded to the host with minimal
> +modification.
> +
> +The crosvm book provides directions on how to build a `gfxstream-enabled
> +rutabaga`_ and launch a `guest Wayland proxy`_.
> +
> +This device does require host blob support (``hostmem`` field below). The
> +``hostmem`` field specifies the size of virtio-gpu host memory window.
> +This is typically between 256M and 8G.
> +
> +At least one capset (see colon separated ``capset_names`` below) must be
> +specified when starting the device.  The currently supported
> +``capset_names`` are ``gfxstream-vulkan`` and ``cross-domain`` on Linux
> +guests. For Android guests, ``gfxstream-gles`` is also supported.
> +
> +The device will try to auto-detect the wayland socket path if the
> +``cross-domain`` capset name is set.  The user may optionally specify
> +``wayland_socket_path`` for non-standard paths.
> +
> +The ``wsi`` option can be set to ``surfaceless`` or ``headless``.
> +Surfaceless doesn't create a native window surface, but does copy from the
> +render target to the Pixman buffer if a virtio-gpu 2D hypercall is issued.
> +Headless is like surfaceless, but doesn't copy to the Pixman buffer.
> +Surfaceless is the default if ``wsi`` is not specified.
> +
> +.. parsed-literal::
> +    -device virtio-gpu-rutabaga,capset_names=gfxstream-vulkan:cross-domain,
> +       hostmem=8G,wayland_socket_path=/tmp/nonstandard/mock_wayland.sock,
> +       wsi=headless
> +
> +.. _rutabaga_gfx: https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h
> +.. _gfxstream: https://android.googlesource.com/platform/hardware/google/gfxstream/
> +.. _Wayland display passthrough: https://www.youtube.com/watch?v=OZJiHMtIQ2M
> +.. _gfxstream-enabled rutabaga: https://crosvm.dev/book/appendix/rutabaga_gfx.html

You have different links for "rutabaga_gfx" and "gfxstream-enabled 
rutabaga", but I think you only need one.

> +.. _guest Wayland proxy: https://crosvm.dev/book/devices/wayland.html

Gurchetan Singh Aug. 17, 2023, 11:47 p.m. UTC | #2

On Wed, Aug 16, 2023 at 10:28 PM Akihiko Odaki <[e-mail]>
wrote:

> On 2023/08/17 11:23, Gurchetan Singh wrote:
> > From: Gurchetan Singh <[e-mail]>
> >
> > This adds basic documentation for virtio-gpu.
> >
> > Suggested-by: Akihiko Odaki <[e-mail]>
> > Signed-off-by: Gurchetan Singh <[e-mail]>
> > Tested-by: Alyssa Ross <[e-mail]>
> > Tested-by: Emmanouil Pitsidianakis <[e-mail]>
> > Reviewed-by: Emmanouil Pitsidianakis <[e-mail]>
> > ---
> > v2: - Incorporated suggestions by Akihiko Odaki
> >      - Listed the currently supported capset_names (Bernard)
> >
> > v3: - Incorporated suggestions by Akihiko Odaki and Alyssa Ross
> >
> > v4: - Incorporated suggestions by Akihiko Odaki
> >
> > v5: - Removed pci suffix from examples
> >      - Verified that -device virtio-gpu-rutabaga works.  Strangely
> >        enough, I don't remember changing anything, and I remember
> >        it not working.  I did rebase to top of tree though.
> >      - Fixed meson examples in crosvm docs
> >
> >   docs/system/device-emulation.rst   |   1 +
> >   docs/system/devices/virtio-gpu.rst | 113 +++++++++++++++++++++++++++++
> >   2 files changed, 114 insertions(+)
> >   create mode 100644 docs/system/devices/virtio-gpu.rst
> >
> > diff --git a/docs/system/device-emulation.rst
> b/docs/system/device-emulation.rst
> > index 4491c4cbf7..1167f3a9f2 100644
> > --- a/docs/system/device-emulation.rst
> > +++ b/docs/system/device-emulation.rst
> > @@ -91,6 +91,7 @@ Emulated Devices
> >      devices/nvme.rst
> >      devices/usb.rst
> >      devices/vhost-user.rst
> > +   devices/virtio-gpu.rst
> >      devices/virtio-pmem.rst
> >      devices/vhost-user-rng.rst
> >      devices/canokey.rst
> > diff --git a/docs/system/devices/virtio-gpu.rst
> b/docs/system/devices/virtio-gpu.rst
> > new file mode 100644
> > index 0000000000..8c5c708272
> > --- /dev/null
> > +++ b/docs/system/devices/virtio-gpu.rst
> > @@ -0,0 +1,113 @@
> > +..
> > +   SPDX-License-Identifier: GPL-2.0
> > +
> > +virtio-gpu
> > +==========
> > +
> > +This document explains the setup and usage of the virtio-gpu device.
> > +The virtio-gpu device paravirtualizes the GPU and display controller.
> > +
> > +Linux kernel support
> > +--------------------
> > +
> > +virtio-gpu requires a guest Linux kernel built with the
> > +``CONFIG_DRM_VIRTIO_GPU`` option.
> > +
> > +QEMU virtio-gpu variants
> > +------------------------
> > +
> > +QEMU virtio-gpu device variants come in the following form:
> > +
> > + * ``virtio-vga[-BACKEND]``
> > + * ``virtio-gpu[-BACKEND][-INTERFACE]``
> > + * ``vhost-user-vga``
> > + * ``vhost-user-pci``
> > +
> > +**Backends:** QEMU provides a 2D virtio-gpu backend, and two accelerated
> > +backends: virglrenderer ('gl' device label) and rutabaga_gfx ('rutabaga'
> > +device label).  There is a vhost-user backend that runs the graphics
> stack
> > +in a separate process for improved isolation.
> > +
> > +**Interfaces:** QEMU further categorizes virtio-gpu device variants
> based
> > +on the interface exposed to the guest. The interfaces can be classified
> > +into VGA and non-VGA variants. The VGA ones are prefixed with virtio-vga
> > +or vhost-user-vga while the non-VGA ones are prefixed with virtio-gpu or
> > +vhost-user-gpu.
> > +
> > +The VGA ones always use the PCI interface, but for the non-VGA ones, the
> > +user can further pick between MMIO or PCI. For MMIO, the user can suffix
> > +the device name with -device, though vhost-user-gpu does not support
> MMIO.
> > +For PCI, the user can suffix it with -pci. Without these suffixes, the
> > +platform default will be chosen.
> > +
> > +virtio-gpu 2d
> > +-------------
> > +
> > +The default 2D backend only performs 2D operations. The guest needs to
> > +employ a software renderer for 3D graphics.
> > +
> > +Typically, the software renderer is provided by `Mesa`_ or
> `SwiftShader`_.
> > +Mesa's implementations (LLVMpipe, Lavapipe and virgl below) work out of
> box
> > +on typical modern Linux distributions.
> > +
> > +.. parsed-literal::
> > +    -device virtio-gpu
> > +
> > +.. _Mesa: https://www.mesa3d.org/
> > +.. _SwiftShader: https://github.com/google/swiftshader
> > +
> > +virtio-gpu virglrenderer
> > +------------------------
> > +
> > +When using virgl accelerated graphics mode in the guest, OpenGL API
> calls
> > +are translated into an intermediate representation (see `Gallium3D`_).
> The
> > +intermediate representation is communicated to the host and the
> > +`virglrenderer`_ library on the host translates the intermediate
> > +representation back to OpenGL API calls.
> > +
> > +.. parsed-literal::
> > +    -device virtio-gpu-gl
> > +
> > +.. _Gallium3D: https://www.freedesktop.org/wiki/Software/gallium/
> > +.. _virglrenderer: https://gitlab.freedesktop.org/virgl/virglrenderer/
> > +
> > +virtio-gpu rutabaga
> > +-------------------
> > +
> > +virtio-gpu can also leverage `rutabaga_gfx`_ to provide `gfxstream`_
> > +rendering and `Wayland display passthrough`_.  With the gfxstream
> rendering
> > +mode, GLES and Vulkan calls are forwarded to the host with minimal
> > +modification.
> > +
> > +The crosvm book provides directions on how to build a `gfxstream-enabled
> > +rutabaga`_ and launch a `guest Wayland proxy`_.
> > +
> > +This device does require host blob support (``hostmem`` field below).
> The
> > +``hostmem`` field specifies the size of virtio-gpu host memory window.
> > +This is typically between 256M and 8G.
> > +
> > +At least one capset (see colon separated ``capset_names`` below) must be
> > +specified when starting the device.  The currently supported
> > +``capset_names`` are ``gfxstream-vulkan`` and ``cross-domain`` on Linux
> > +guests. For Android guests, ``gfxstream-gles`` is also supported.
> > +
> > +The device will try to auto-detect the wayland socket path if the
> > +``cross-domain`` capset name is set.  The user may optionally specify
> > +``wayland_socket_path`` for non-standard paths.
> > +
> > +The ``wsi`` option can be set to ``surfaceless`` or ``headless``.
> > +Surfaceless doesn't create a native window surface, but does copy from
> the
> > +render target to the Pixman buffer if a virtio-gpu 2D hypercall is
> issued.
> > +Headless is like surfaceless, but doesn't copy to the Pixman buffer.
> > +Surfaceless is the default if ``wsi`` is not specified.
> > +
> > +.. parsed-literal::
> > +    -device
> virtio-gpu-rutabaga,capset_names=gfxstream-vulkan:cross-domain,
> > +
>  hostmem=8G,wayland_socket_path=/tmp/nonstandard/mock_wayland.sock,
> > +       wsi=headless
> > +
> > +.. _rutabaga_gfx:
> https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h
> > +.. _gfxstream:
> https://android.googlesource.com/platform/hardware/google/gfxstream/
> > +.. _Wayland display passthrough:
> https://www.youtube.com/watch?v=OZJiHMtIQ2M
> > +.. _gfxstream-enabled rutabaga:
> https://crosvm.dev/book/appendix/rutabaga_gfx.html
>
> You have different links for "rutabaga_gfx" and "gfxstream-enabled
> rutabaga", but I think you only need one.
>

Done.  Didn't resend the entire patch series (to avoid spamming list), just
did "in-reply-to".  The change is also available at:

https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8


>
> > +.. _guest Wayland proxy: https://crosvm.dev/book/devices/wayland.html
>

Akihiko Odaki Aug. 18, 2023, 12:08 p.m. UTC | #3

On 2023/08/18 8:47, Gurchetan Singh wrote:
> 
> 
> On Wed, Aug 16, 2023 at 10:28 PM Akihiko Odaki <[e-mail] 
> <mailto:[e-mail]>> wrote:
> 
>     On 2023/08/17 11:23, Gurchetan Singh wrote:
>      > From: Gurchetan Singh <[e-mail]
>     <mailto:[e-mail]>>
>      >
>      > This adds basic documentation for virtio-gpu.
>      >
>      > Suggested-by: Akihiko Odaki <[e-mail]
>     <mailto:[e-mail]>>
>      > Signed-off-by: Gurchetan Singh <[e-mail]
>     <mailto:[e-mail]>>
>      > Tested-by: Alyssa Ross <[e-mail] <mailto:[e-mail]>>
>      > Tested-by: Emmanouil Pitsidianakis
>     <[e-mail] <mailto:[e-mail]>>
>      > Reviewed-by: Emmanouil Pitsidianakis
>     <[e-mail] <mailto:[e-mail]>>
>      > ---
>      > v2: - Incorporated suggestions by Akihiko Odaki
>      >      - Listed the currently supported capset_names (Bernard)
>      >
>      > v3: - Incorporated suggestions by Akihiko Odaki and Alyssa Ross
>      >
>      > v4: - Incorporated suggestions by Akihiko Odaki
>      >
>      > v5: - Removed pci suffix from examples
>      >      - Verified that -device virtio-gpu-rutabaga works.  Strangely
>      >        enough, I don't remember changing anything, and I remember
>      >        it not working.  I did rebase to top of tree though.
>      >      - Fixed meson examples in crosvm docs
>      >
>      >   docs/system/device-emulation.rst   |   1 +
>      >   docs/system/devices/virtio-gpu.rst | 113
>     +++++++++++++++++++++++++++++
>      >   2 files changed, 114 insertions(+)
>      >   create mode 100644 docs/system/devices/virtio-gpu.rst
>      >
>      > diff --git a/docs/system/device-emulation.rst
>     b/docs/system/device-emulation.rst
>      > index 4491c4cbf7..1167f3a9f2 100644
>      > --- a/docs/system/device-emulation.rst
>      > +++ b/docs/system/device-emulation.rst
>      > @@ -91,6 +91,7 @@ Emulated Devices
>      >      devices/nvme.rst
>      >      devices/usb.rst
>      >      devices/vhost-user.rst
>      > +   devices/virtio-gpu.rst
>      >      devices/virtio-pmem.rst
>      >      devices/vhost-user-rng.rst
>      >      devices/canokey.rst
>      > diff --git a/docs/system/devices/virtio-gpu.rst
>     b/docs/system/devices/virtio-gpu.rst
>      > new file mode 100644
>      > index 0000000000..8c5c708272
>      > --- /dev/null
>      > +++ b/docs/system/devices/virtio-gpu.rst
>      > @@ -0,0 +1,113 @@
>      > +..
>      > +   SPDX-License-Identifier: GPL-2.0
>      > +
>      > +virtio-gpu
>      > +==========
>      > +
>      > +This document explains the setup and usage of the virtio-gpu device.
>      > +The virtio-gpu device paravirtualizes the GPU and display
>     controller.
>      > +
>      > +Linux kernel support
>      > +--------------------
>      > +
>      > +virtio-gpu requires a guest Linux kernel built with the
>      > +``CONFIG_DRM_VIRTIO_GPU`` option.
>      > +
>      > +QEMU virtio-gpu variants
>      > +------------------------
>      > +
>      > +QEMU virtio-gpu device variants come in the following form:
>      > +
>      > + * ``virtio-vga[-BACKEND]``
>      > + * ``virtio-gpu[-BACKEND][-INTERFACE]``
>      > + * ``vhost-user-vga``
>      > + * ``vhost-user-pci``
>      > +
>      > +**Backends:** QEMU provides a 2D virtio-gpu backend, and two
>     accelerated
>      > +backends: virglrenderer ('gl' device label) and rutabaga_gfx
>     ('rutabaga'
>      > +device label).  There is a vhost-user backend that runs the
>     graphics stack
>      > +in a separate process for improved isolation.
>      > +
>      > +**Interfaces:** QEMU further categorizes virtio-gpu device
>     variants based
>      > +on the interface exposed to the guest. The interfaces can be
>     classified
>      > +into VGA and non-VGA variants. The VGA ones are prefixed with
>     virtio-vga
>      > +or vhost-user-vga while the non-VGA ones are prefixed with
>     virtio-gpu or
>      > +vhost-user-gpu.
>      > +
>      > +The VGA ones always use the PCI interface, but for the non-VGA
>     ones, the
>      > +user can further pick between MMIO or PCI. For MMIO, the user
>     can suffix
>      > +the device name with -device, though vhost-user-gpu does not
>     support MMIO.
>      > +For PCI, the user can suffix it with -pci. Without these
>     suffixes, the
>      > +platform default will be chosen.
>      > +
>      > +virtio-gpu 2d
>      > +-------------
>      > +
>      > +The default 2D backend only performs 2D operations. The guest
>     needs to
>      > +employ a software renderer for 3D graphics.
>      > +
>      > +Typically, the software renderer is provided by `Mesa`_ or
>     `SwiftShader`_.
>      > +Mesa's implementations (LLVMpipe, Lavapipe and virgl below) work
>     out of box
>      > +on typical modern Linux distributions.
>      > +
>      > +.. parsed-literal::
>      > +    -device virtio-gpu
>      > +
>      > +.. _Mesa: https://www.mesa3d.org/ <https://www.mesa3d.org/>
>      > +.. _SwiftShader: https://github.com/google/swiftshader
>     <https://github.com/google/swiftshader>
>      > +
>      > +virtio-gpu virglrenderer
>      > +------------------------
>      > +
>      > +When using virgl accelerated graphics mode in the guest, OpenGL
>     API calls
>      > +are translated into an intermediate representation (see
>     `Gallium3D`_). The
>      > +intermediate representation is communicated to the host and the
>      > +`virglrenderer`_ library on the host translates the intermediate
>      > +representation back to OpenGL API calls.
>      > +
>      > +.. parsed-literal::
>      > +    -device virtio-gpu-gl
>      > +
>      > +.. _Gallium3D:
>     https://www.freedesktop.org/wiki/Software/gallium/
>     <https://www.freedesktop.org/wiki/Software/gallium/>
>      > +.. _virglrenderer:
>     https://gitlab.freedesktop.org/virgl/virglrenderer/
>     <https://gitlab.freedesktop.org/virgl/virglrenderer/>
>      > +
>      > +virtio-gpu rutabaga
>      > +-------------------
>      > +
>      > +virtio-gpu can also leverage `rutabaga_gfx`_ to provide `gfxstream`_
>      > +rendering and `Wayland display passthrough`_.  With the
>     gfxstream rendering
>      > +mode, GLES and Vulkan calls are forwarded to the host with minimal
>      > +modification.
>      > +
>      > +The crosvm book provides directions on how to build a
>     `gfxstream-enabled
>      > +rutabaga`_ and launch a `guest Wayland proxy`_.
>      > +
>      > +This device does require host blob support (``hostmem`` field
>     below). The
>      > +``hostmem`` field specifies the size of virtio-gpu host memory
>     window.
>      > +This is typically between 256M and 8G.
>      > +
>      > +At least one capset (see colon separated ``capset_names`` below)
>     must be
>      > +specified when starting the device.  The currently supported
>      > +``capset_names`` are ``gfxstream-vulkan`` and ``cross-domain``
>     on Linux
>      > +guests. For Android guests, ``gfxstream-gles`` is also supported.
>      > +
>      > +The device will try to auto-detect the wayland socket path if the
>      > +``cross-domain`` capset name is set.  The user may optionally
>     specify
>      > +``wayland_socket_path`` for non-standard paths.
>      > +
>      > +The ``wsi`` option can be set to ``surfaceless`` or ``headless``.
>      > +Surfaceless doesn't create a native window surface, but does
>     copy from the
>      > +render target to the Pixman buffer if a virtio-gpu 2D hypercall
>     is issued.
>      > +Headless is like surfaceless, but doesn't copy to the Pixman buffer.
>      > +Surfaceless is the default if ``wsi`` is not specified.
>      > +
>      > +.. parsed-literal::
>      > +    -device
>     virtio-gpu-rutabaga,capset_names=gfxstream-vulkan:cross-domain,
>      > +     
>       hostmem=8G,wayland_socket_path=/tmp/nonstandard/mock_wayland.sock,
>      > +       wsi=headless
>      > +
>      > +.. _rutabaga_gfx:
>     https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h <https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h>
>      > +.. _gfxstream:
>     https://android.googlesource.com/platform/hardware/google/gfxstream/
>     <https://android.googlesource.com/platform/hardware/google/gfxstream/>
>      > +.. _Wayland display passthrough:
>     https://www.youtube.com/watch?v=OZJiHMtIQ2M
>     <https://www.youtube.com/watch?v=OZJiHMtIQ2M>
>      > +.. _gfxstream-enabled rutabaga:
>     https://crosvm.dev/book/appendix/rutabaga_gfx.html
>     <https://crosvm.dev/book/appendix/rutabaga_gfx.html>
> 
>     You have different links for "rutabaga_gfx" and "gfxstream-enabled
>     rutabaga", but I think you only need one.
> 
> 
> Done.  Didn't resend the entire patch series (to avoid spamming list), 
> just did "in-reply-to".  The change is also available at:
> 
> https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8 <https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8>

The patch series now looks good so I finally decided to try this patch 
series, but I couldn't get it work.

I noticed gfxstream has page size hardcoded as 4 KiB, which broke my 
setup on M2 MacBook Air (running Asahi Linux) that has 16 KiB page. You 
may add code to check host page size and to report an error if it's not 
4 KiB to virtio-gpu-rutabaga, but I think it's trivial to fix gfxstream 
to query page size at runtime as QEMU and Rutabaga does so I hope you to 
do so. For testing purpose, I have replaced it with 16 KiB on my setup.

I also found some bugs on QEMU side so I added comments to respective 
patches.

Below is the logs from my last attempt of running vkgears:

$ VK_LOADER_DEBUG=all demos/build/src/vulkan/vkgears
INFO:             Vulkan Loader Version 1.3.243
LAYER:            Searching for layer manifest files
LAYER:               In following locations:
LAYER:                  /var/home/person/.config/vulkan/implicit_layer.d
LAYER:                  /etc/xdg/vulkan/implicit_layer.d
LAYER:                  /etc/vulkan/implicit_layer.d
LAYER: 
/var/home/person/.local/share/vulkan/implicit_layer.d
LAYER: 
/var/home/person/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
LAYER: 
/var/lib/flatpak/exports/share/vulkan/implicit_layer.d
LAYER:                  /usr/local/share/vulkan/implicit_layer.d
LAYER:                  /usr/share/vulkan/implicit_layer.d
LAYER:               Found the following files:
LAYER: 
/usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
INFO:             Found manifest file 
/usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file 
version "1.0.0")
LAYER:            Searching for layer manifest files
LAYER:               In following locations:
LAYER:                  /var/home/person/.config/vulkan/explicit_layer.d
LAYER:                  /etc/xdg/vulkan/explicit_layer.d
LAYER:                  /etc/vulkan/explicit_layer.d
LAYER: 
/var/home/person/.local/share/vulkan/explicit_layer.d
LAYER: 
/var/home/person/.local/share/flatpak/exports/share/vulkan/explicit_layer.d
LAYER: 
/var/lib/flatpak/exports/share/vulkan/explicit_layer.d
LAYER:                  /usr/local/share/vulkan/explicit_layer.d
LAYER:                  /usr/share/vulkan/explicit_layer.d
LAYER:               Found no files
DRIVER:           Searching for driver manifest files
DRIVER:              In following locations:
DRIVER:                 /var/home/person/.config/vulkan/icd.d
DRIVER:                 /etc/xdg/vulkan/icd.d
DRIVER:                 /etc/vulkan/icd.d
DRIVER:                 /var/home/person/.local/share/vulkan/icd.d
DRIVER: 
/var/home/person/.local/share/flatpak/exports/share/vulkan/icd.d
DRIVER:                 /var/lib/flatpak/exports/share/vulkan/icd.d
DRIVER:                 /usr/local/share/vulkan/icd.d
DRIVER:                 /usr/share/vulkan/icd.d
DRIVER:              Found the following files:
DRIVER: 
/usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json
DRIVER:                 /usr/share/vulkan/icd.d/broadcom_icd.aarch64.json
DRIVER:                 /usr/share/vulkan/icd.d/freedreno_icd.aarch64.json
DRIVER:                 /usr/share/vulkan/icd.d/lvp_icd.aarch64.json
DRIVER:                 /usr/share/vulkan/icd.d/panfrost_icd.aarch64.json
DRIVER:                 /usr/share/vulkan/icd.d/radeon_icd.aarch64.json
DRIVER:           Found ICD manifest file 
/usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/local/lib64/libvulkan_cereal.so
[VirtGpuDevice.cpp(71)]
[prio 6] virtgpu backend not enabling 
VIRTGPU_PARAM_CREATE_GUEST_HANDLE[AndroidHealthMonitor.cpp(275)]
[prio 4] HealthMonitor disabled. Returning nullptrI0818 21:00:07.980257 
154451 IntelDrmDecoder.cpp:38] IntelDrmDecoder created for context 2
DRIVER:           Found ICD manifest file 
/usr/share/vulkan/icd.d/broadcom_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/lib64/libvulkan_broadcom.so
DRIVER:           Found ICD manifest file 
/usr/share/vulkan/icd.d/freedreno_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/lib64/libvulkan_freedreno.so
DRIVER:           Found ICD manifest file 
/usr/share/vulkan/icd.d/lvp_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/lib64/libvulkan_lvp.so
DRIVER:           Found ICD manifest file 
/usr/share/vulkan/icd.d/panfrost_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/lib64/libvulkan_panfrost.so
DRIVER:           Found ICD manifest file 
/usr/share/vulkan/icd.d/radeon_icd.aarch64.json, version "1.0.0"
DEBUG | DRIVER:   Searching for ICD drivers named 
/usr/lib64/libvulkan_radeon.so
DEBUG | LAYER:    Loading layer library libVkLayer_MESA_device_select.so
INFO | LAYER:     Insert instance layer "VK_LAYER_MESA_device_select" 
(libVkLayer_MESA_device_select.so)
LAYER:            vkCreateInstance layer callstack setup to:
LAYER:               <Application>
LAYER:                 ||
LAYER:               <Loader>
LAYER:                 ||
LAYER:               VK_LAYER_MESA_device_select
LAYER:                       Type: Implicit
LAYER:                           Disable Env Var:  NODEVICE_SELECT
LAYER:                       Manifest: 
/usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
LAYER:                       Library:  libVkLayer_MESA_device_select.so
LAYER:                 ||
LAYER:               <Drivers>
I0818 21:00:08.014896  154451 VkDecoderGlobalState.cpp:443] Creating 
Vulkan instance for app: vkgears
INFO | DRIVER:    linux_read_sorted_physical_devices:
INFO | DRIVER:         Original order:
INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:         Sorted order:
INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
INFO | DRIVER:    linux_read_sorted_physical_devices:
INFO | DRIVER:         Original order:
INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:         Sorted order:
INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
DEBUG | DRIVER:   Copying old device 0 into new device 0
DEBUG | DRIVER:   Copying old device 1 into new device 1
INFO | DRIVER:    linux_read_sorted_physical_devices:
INFO | DRIVER:         Original order:
INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:         Sorted order:
INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
DEBUG | DRIVER:   Copying old device 0 into new device 0
DEBUG | DRIVER:   Copying old device 1 into new device 1
INFO | DRIVER:    linux_read_sorted_physical_devices:
INFO | DRIVER:         Original order:
INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:         Sorted order:
INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
DEBUG | DRIVER:   Copying old device 0 into new device 0
DEBUG | DRIVER:   Copying old device 1 into new device 1
ERROR:            loader_validate_device_extensions: Device extension 
VK_KHR_swapchain not supported by selected physical device or enabled 
layers.
ERROR:            vkCreateDevice: Failed to validate extensions in list
ERROR:            vkGetDeviceQueue2: Invalid device 
[VUID-vkGetDeviceQueue2-device-parameter]
Aborted (core dumped)

Gurchetan Singh Aug. 19, 2023, 1:17 a.m. UTC | #4

On Fri, Aug 18, 2023 at 5:08 AM Akihiko Odaki <[e-mail]>
wrote:

> On 2023/08/18 8:47, Gurchetan Singh wrote:
> >
> >
> > On Wed, Aug 16, 2023 at 10:28 PM Akihiko Odaki <[e-mail]
> > <mailto:[e-mail]>> wrote:
> >
> >     On 2023/08/17 11:23, Gurchetan Singh wrote:
> >      > From: Gurchetan Singh <[e-mail]
> >     <mailto:[e-mail]>>
> >      >
> >      > This adds basic documentation for virtio-gpu.
> >      >
> >      > Suggested-by: Akihiko Odaki <[e-mail]
> >     <mailto:[e-mail]>>
> >      > Signed-off-by: Gurchetan Singh <[e-mail]
> >     <mailto:[e-mail]>>
> >      > Tested-by: Alyssa Ross <[e-mail] <mailto:[e-mail]>>
> >      > Tested-by: Emmanouil Pitsidianakis
> >     <[e-mail] <mailto:
> [e-mail]>>
> >      > Reviewed-by: Emmanouil Pitsidianakis
> >     <[e-mail] <mailto:
> [e-mail]>>
> >      > ---
> >      > v2: - Incorporated suggestions by Akihiko Odaki
> >      >      - Listed the currently supported capset_names (Bernard)
> >      >
> >      > v3: - Incorporated suggestions by Akihiko Odaki and Alyssa Ross
> >      >
> >      > v4: - Incorporated suggestions by Akihiko Odaki
> >      >
> >      > v5: - Removed pci suffix from examples
> >      >      - Verified that -device virtio-gpu-rutabaga works.  Strangely
> >      >        enough, I don't remember changing anything, and I remember
> >      >        it not working.  I did rebase to top of tree though.
> >      >      - Fixed meson examples in crosvm docs
> >      >
> >      >   docs/system/device-emulation.rst   |   1 +
> >      >   docs/system/devices/virtio-gpu.rst | 113
> >     +++++++++++++++++++++++++++++
> >      >   2 files changed, 114 insertions(+)
> >      >   create mode 100644 docs/system/devices/virtio-gpu.rst
> >      >
> >      > diff --git a/docs/system/device-emulation.rst
> >     b/docs/system/device-emulation.rst
> >      > index 4491c4cbf7..1167f3a9f2 100644
> >      > --- a/docs/system/device-emulation.rst
> >      > +++ b/docs/system/device-emulation.rst
> >      > @@ -91,6 +91,7 @@ Emulated Devices
> >      >      devices/nvme.rst
> >      >      devices/usb.rst
> >      >      devices/vhost-user.rst
> >      > +   devices/virtio-gpu.rst
> >      >      devices/virtio-pmem.rst
> >      >      devices/vhost-user-rng.rst
> >      >      devices/canokey.rst
> >      > diff --git a/docs/system/devices/virtio-gpu.rst
> >     b/docs/system/devices/virtio-gpu.rst
> >      > new file mode 100644
> >      > index 0000000000..8c5c708272
> >      > --- /dev/null
> >      > +++ b/docs/system/devices/virtio-gpu.rst
> >      > @@ -0,0 +1,113 @@
> >      > +..
> >      > +   SPDX-License-Identifier: GPL-2.0
> >      > +
> >      > +virtio-gpu
> >      > +==========
> >      > +
> >      > +This document explains the setup and usage of the virtio-gpu
> device.
> >      > +The virtio-gpu device paravirtualizes the GPU and display
> >     controller.
> >      > +
> >      > +Linux kernel support
> >      > +--------------------
> >      > +
> >      > +virtio-gpu requires a guest Linux kernel built with the
> >      > +``CONFIG_DRM_VIRTIO_GPU`` option.
> >      > +
> >      > +QEMU virtio-gpu variants
> >      > +------------------------
> >      > +
> >      > +QEMU virtio-gpu device variants come in the following form:
> >      > +
> >      > + * ``virtio-vga[-BACKEND]``
> >      > + * ``virtio-gpu[-BACKEND][-INTERFACE]``
> >      > + * ``vhost-user-vga``
> >      > + * ``vhost-user-pci``
> >      > +
> >      > +**Backends:** QEMU provides a 2D virtio-gpu backend, and two
> >     accelerated
> >      > +backends: virglrenderer ('gl' device label) and rutabaga_gfx
> >     ('rutabaga'
> >      > +device label).  There is a vhost-user backend that runs the
> >     graphics stack
> >      > +in a separate process for improved isolation.
> >      > +
> >      > +**Interfaces:** QEMU further categorizes virtio-gpu device
> >     variants based
> >      > +on the interface exposed to the guest. The interfaces can be
> >     classified
> >      > +into VGA and non-VGA variants. The VGA ones are prefixed with
> >     virtio-vga
> >      > +or vhost-user-vga while the non-VGA ones are prefixed with
> >     virtio-gpu or
> >      > +vhost-user-gpu.
> >      > +
> >      > +The VGA ones always use the PCI interface, but for the non-VGA
> >     ones, the
> >      > +user can further pick between MMIO or PCI. For MMIO, the user
> >     can suffix
> >      > +the device name with -device, though vhost-user-gpu does not
> >     support MMIO.
> >      > +For PCI, the user can suffix it with -pci. Without these
> >     suffixes, the
> >      > +platform default will be chosen.
> >      > +
> >      > +virtio-gpu 2d
> >      > +-------------
> >      > +
> >      > +The default 2D backend only performs 2D operations. The guest
> >     needs to
> >      > +employ a software renderer for 3D graphics.
> >      > +
> >      > +Typically, the software renderer is provided by `Mesa`_ or
> >     `SwiftShader`_.
> >      > +Mesa's implementations (LLVMpipe, Lavapipe and virgl below) work
> >     out of box
> >      > +on typical modern Linux distributions.
> >      > +
> >      > +.. parsed-literal::
> >      > +    -device virtio-gpu
> >      > +
> >      > +.. _Mesa: https://www.mesa3d.org/ <https://www.mesa3d.org/>
> >      > +.. _SwiftShader: https://github.com/google/swiftshader
> >     <https://github.com/google/swiftshader>
> >      > +
> >      > +virtio-gpu virglrenderer
> >      > +------------------------
> >      > +
> >      > +When using virgl accelerated graphics mode in the guest, OpenGL
> >     API calls
> >      > +are translated into an intermediate representation (see
> >     `Gallium3D`_). The
> >      > +intermediate representation is communicated to the host and the
> >      > +`virglrenderer`_ library on the host translates the intermediate
> >      > +representation back to OpenGL API calls.
> >      > +
> >      > +.. parsed-literal::
> >      > +    -device virtio-gpu-gl
> >      > +
> >      > +.. _Gallium3D:
> >     https://www.freedesktop.org/wiki/Software/gallium/
> >     <https://www.freedesktop.org/wiki/Software/gallium/>
> >      > +.. _virglrenderer:
> >     https://gitlab.freedesktop.org/virgl/virglrenderer/
> >     <https://gitlab.freedesktop.org/virgl/virglrenderer/>
> >      > +
> >      > +virtio-gpu rutabaga
> >      > +-------------------
> >      > +
> >      > +virtio-gpu can also leverage `rutabaga_gfx`_ to provide
> `gfxstream`_
> >      > +rendering and `Wayland display passthrough`_.  With the
> >     gfxstream rendering
> >      > +mode, GLES and Vulkan calls are forwarded to the host with
> minimal
> >      > +modification.
> >      > +
> >      > +The crosvm book provides directions on how to build a
> >     `gfxstream-enabled
> >      > +rutabaga`_ and launch a `guest Wayland proxy`_.
> >      > +
> >      > +This device does require host blob support (``hostmem`` field
> >     below). The
> >      > +``hostmem`` field specifies the size of virtio-gpu host memory
> >     window.
> >      > +This is typically between 256M and 8G.
> >      > +
> >      > +At least one capset (see colon separated ``capset_names`` below)
> >     must be
> >      > +specified when starting the device.  The currently supported
> >      > +``capset_names`` are ``gfxstream-vulkan`` and ``cross-domain``
> >     on Linux
> >      > +guests. For Android guests, ``gfxstream-gles`` is also supported.
> >      > +
> >      > +The device will try to auto-detect the wayland socket path if the
> >      > +``cross-domain`` capset name is set.  The user may optionally
> >     specify
> >      > +``wayland_socket_path`` for non-standard paths.
> >      > +
> >      > +The ``wsi`` option can be set to ``surfaceless`` or ``headless``.
> >      > +Surfaceless doesn't create a native window surface, but does
> >     copy from the
> >      > +render target to the Pixman buffer if a virtio-gpu 2D hypercall
> >     is issued.
> >      > +Headless is like surfaceless, but doesn't copy to the Pixman
> buffer.
> >      > +Surfaceless is the default if ``wsi`` is not specified.
> >      > +
> >      > +.. parsed-literal::
> >      > +    -device
> >     virtio-gpu-rutabaga,capset_names=gfxstream-vulkan:cross-domain,
> >      > +
> >       hostmem=8G,wayland_socket_path=/tmp/nonstandard/mock_wayland.sock,
> >      > +       wsi=headless
> >      > +
> >      > +.. _rutabaga_gfx:
> >
> https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h
> <
> https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h
> >
> >      > +.. _gfxstream:
> >     https://android.googlesource.com/platform/hardware/google/gfxstream/
> >     <
> https://android.googlesource.com/platform/hardware/google/gfxstream/>
> >      > +.. _Wayland display passthrough:
> >     https://www.youtube.com/watch?v=OZJiHMtIQ2M
> >     <https://www.youtube.com/watch?v=OZJiHMtIQ2M>
> >      > +.. _gfxstream-enabled rutabaga:
> >     https://crosvm.dev/book/appendix/rutabaga_gfx.html
> >     <https://crosvm.dev/book/appendix/rutabaga_gfx.html>
> >
> >     You have different links for "rutabaga_gfx" and "gfxstream-enabled
> >     rutabaga", but I think you only need one.
> >
> >
> > Done.  Didn't resend the entire patch series (to avoid spamming list),
> > just did "in-reply-to".  The change is also available at:
> >
> >
> https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8
> <
> https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8
> >
>
> The patch series now looks good so I finally decided to try this patch
> series, but I couldn't get it work.
>
> I noticed gfxstream has page size hardcoded as 4 KiB, which broke my
> setup on M2 MacBook Air (running Asahi Linux) that has 16 KiB page. You
> may add code to check host page size and to report an error if it's not
> 4 KiB to virtio-gpu-rutabaga, but I think it's trivial to fix gfxstream
> to query page size at runtime as QEMU and Rutabaga does so I hope you to
> do so. For testing purpose, I have replaced it with 16 KiB on my setup.
>

Good catch, the fix to the 16KiB was merged today and is in gfxstream ToT
right now.


>
> I also found some bugs on QEMU side so I added comments to respective
> patches.
>
> Below is the logs from my last attempt of running vkgears:
>
> $ VK_LOADER_DEBUG=all demos/build/src/vulkan/vkgears
> INFO:             Vulkan Loader Version 1.3.243
> LAYER:            Searching for layer manifest files
> LAYER:               In following locations:
> LAYER:                  /var/home/person/.config/vulkan/implicit_layer.d
> LAYER:                  /etc/xdg/vulkan/implicit_layer.d
> LAYER:                  /etc/vulkan/implicit_layer.d
> LAYER:
> /var/home/person/.local/share/vulkan/implicit_layer.d
> LAYER:
> /var/home/person/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
> LAYER:
> /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
> LAYER:                  /usr/local/share/vulkan/implicit_layer.d
> LAYER:                  /usr/share/vulkan/implicit_layer.d
> LAYER:               Found the following files:
> LAYER:
> /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
> INFO:             Found manifest file
> /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json (file
> version "1.0.0")
> LAYER:            Searching for layer manifest files
> LAYER:               In following locations:
> LAYER:                  /var/home/person/.config/vulkan/explicit_layer.d
> LAYER:                  /etc/xdg/vulkan/explicit_layer.d
> LAYER:                  /etc/vulkan/explicit_layer.d
> LAYER:
> /var/home/person/.local/share/vulkan/explicit_layer.d
> LAYER:
> /var/home/person/.local/share/flatpak/exports/share/vulkan/explicit_layer.d
> LAYER:
> /var/lib/flatpak/exports/share/vulkan/explicit_layer.d
> LAYER:                  /usr/local/share/vulkan/explicit_layer.d
> LAYER:                  /usr/share/vulkan/explicit_layer.d
> LAYER:               Found no files
> DRIVER:           Searching for driver manifest files
> DRIVER:              In following locations:
> DRIVER:                 /var/home/person/.config/vulkan/icd.d
> DRIVER:                 /etc/xdg/vulkan/icd.d
> DRIVER:                 /etc/vulkan/icd.d
> DRIVER:                 /var/home/person/.local/share/vulkan/icd.d
> DRIVER:
> /var/home/person/.local/share/flatpak/exports/share/vulkan/icd.d
> DRIVER:                 /var/lib/flatpak/exports/share/vulkan/icd.d
> DRIVER:                 /usr/local/share/vulkan/icd.d
> DRIVER:                 /usr/share/vulkan/icd.d
> DRIVER:              Found the following files:
> DRIVER:
> /usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json
> DRIVER:                 /usr/share/vulkan/icd.d/broadcom_icd.aarch64.json
> DRIVER:                 /usr/share/vulkan/icd.d/freedreno_icd.aarch64.json
> DRIVER:                 /usr/share/vulkan/icd.d/lvp_icd.aarch64.json
> DRIVER:                 /usr/share/vulkan/icd.d/panfrost_icd.aarch64.json
> DRIVER:                 /usr/share/vulkan/icd.d/radeon_icd.aarch64.json
> DRIVER:           Found ICD manifest file
> /usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json, version "1.0.0"
> DEBUG | DRIVER:   Searching for ICD drivers named
> /usr/local/lib64/libvulkan_cereal.so
> [VirtGpuDevice.cpp(71)]
> [prio 6] virtgpu backend not enabling
> VIRTGPU_PARAM_CREATE_GUEST_HANDLE[AndroidHealthMonitor.cpp(275)]
> [prio 4] HealthMonitor disabled. Returning nullptrI0818 21:00:07.980257
> 154451 IntelDrmDecoder.cpp:38] IntelDrmDecoder created for context 2
> DRIVER:           Found ICD manifest file
> /usr/share/vulkan/icd.d/broadcom_icd.aarch64.json, version "1.0.0"
> DEBUG | DRIVER:   Searching for ICD drivers named
> /usr/lib64/libvulkan_broadcom.so
> DRIVER:           Found ICD manifest file
> /usr/share/vulkan/icd.d/freedreno_icd.aarch64.json, version "1.0.0"
> DEBUG | DRIVER:   Searching for ICD drivers named
> /usr/lib64/libvulkan_freedreno.so
> DRIVER:           Found ICD manifest file
> /usr/share/vulkan/icd.d/lvp_icd.aarch64.json, version "1.0.0"
> DEBUG | DRIVER:   Searching for ICD drivers named
> /usr/lib64/libvulkan_lvp.so
> DRIVER:           Found ICD manifest file
> /usr/share/vulkan/icd.d/panfrost_icd.aarch64.json, version "1.0.0"
> DEBUG | DRIVER:   Searching for ICD drivers named
> /usr/lib64/libvulkan_panfrost.so
> DRIVER:           Found ICD manifest file
> /usr/share/vulkan/icd.d/radeon_icd.aarch64.json, version "1.0.0"
> DEBUG | DRIVER:   Searching for ICD drivers named
> /usr/lib64/libvulkan_radeon.so
> DEBUG | LAYER:    Loading layer library libVkLayer_MESA_device_select.so
> INFO | LAYER:     Insert instance layer "VK_LAYER_MESA_device_select"
> (libVkLayer_MESA_device_select.so)
> LAYER:            vkCreateInstance layer callstack setup to:
> LAYER:               <Application>
> LAYER:                 ||
> LAYER:               <Loader>
> LAYER:                 ||
> LAYER:               VK_LAYER_MESA_device_select
> LAYER:                       Type: Implicit
> LAYER:                           Disable Env Var:  NODEVICE_SELECT
> LAYER:                       Manifest:
> /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
> LAYER:                       Library:  libVkLayer_MESA_device_select.so
> LAYER:                 ||
> LAYER:               <Drivers>
> I0818 21:00:08.014896  154451 VkDecoderGlobalState.cpp:443] Creating
> Vulkan instance for app: vkgears
> INFO | DRIVER:    linux_read_sorted_physical_devices:
> INFO | DRIVER:         Original order:
> INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
> INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
> INFO | DRIVER:         Sorted order:
> INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
> INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
> INFO | DRIVER:    linux_read_sorted_physical_devices:
> INFO | DRIVER:         Original order:
> INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
> INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
> INFO | DRIVER:         Sorted order:
> INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
> INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
> DEBUG | DRIVER:   Copying old device 0 into new device 0
> DEBUG | DRIVER:   Copying old device 1 into new device 1
> INFO | DRIVER:    linux_read_sorted_physical_devices:
> INFO | DRIVER:         Original order:
> INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
> INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
> INFO | DRIVER:         Sorted order:
> INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
> INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
> DEBUG | DRIVER:   Copying old device 0 into new device 0
> DEBUG | DRIVER:   Copying old device 1 into new device 1
> INFO | DRIVER:    linux_read_sorted_physical_devices:
> INFO | DRIVER:         Original order:
> INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
> INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
> INFO | DRIVER:         Sorted order:
> INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
> INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
> DEBUG | DRIVER:   Copying old device 0 into new device 0
> DEBUG | DRIVER:   Copying old device 1 into new device 1
> ERROR:            loader_validate_device_extensions: Device extension
> VK_KHR_swapchain not supported by selected physical device or enabled


Yeah, any non-headless Linux tests are unlikely to work.  Maybe in a future
gfxstream release, since our focus is of course on Android and getting
Vulkan in QEMU in general.


>
> layers.
> ERROR:            vkCreateDevice: Failed to validate extensions in list
> ERROR:            vkGetDeviceQueue2: Invalid device
> [VUID-vkGetDeviceQueue2-device-parameter]
> Aborted (core dumped)
>

Akihiko Odaki Aug. 19, 2023, 6:13 a.m. UTC | #5

On 2023/08/19 10:17, Gurchetan Singh wrote:
> 
> 
> On Fri, Aug 18, 2023 at 5:08 AM Akihiko Odaki <[e-mail] 
> <mailto:[e-mail]>> wrote:
> 
>     On 2023/08/18 8:47, Gurchetan Singh wrote:
>      >
>      >
>      > On Wed, Aug 16, 2023 at 10:28 PM Akihiko Odaki
>     <[e-mail] <mailto:[e-mail]>
>      > <mailto:[e-mail]
>     <mailto:[e-mail]>>> wrote:
>      >
>      >     On 2023/08/17 11:23, Gurchetan Singh wrote:
>      >      > From: Gurchetan Singh <[e-mail]
>     <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>     <mailto:[e-mail]>>>
>      >      >
>      >      > This adds basic documentation for virtio-gpu.
>      >      >
>      >      > Suggested-by: Akihiko Odaki <[e-mail]
>     <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>     <mailto:[e-mail]>>>
>      >      > Signed-off-by: Gurchetan Singh
>     <[e-mail] <mailto:[e-mail]>
>      >     <mailto:[e-mail]
>     <mailto:[e-mail]>>>
>      >      > Tested-by: Alyssa Ross <[e-mail] <mailto:[e-mail]>
>     <mailto:[e-mail] <mailto:[e-mail]>>>
>      >      > Tested-by: Emmanouil Pitsidianakis
>      >     <[e-mail]
>     <mailto:[e-mail]>
>     <mailto:[e-mail]
>     <mailto:[e-mail]>>>
>      >      > Reviewed-by: Emmanouil Pitsidianakis
>      >     <[e-mail]
>     <mailto:[e-mail]>
>     <mailto:[e-mail]
>     <mailto:[e-mail]>>>
>      >      > ---
>      >      > v2: - Incorporated suggestions by Akihiko Odaki
>      >      >      - Listed the currently supported capset_names (Bernard)
>      >      >
>      >      > v3: - Incorporated suggestions by Akihiko Odaki and Alyssa
>     Ross
>      >      >
>      >      > v4: - Incorporated suggestions by Akihiko Odaki
>      >      >
>      >      > v5: - Removed pci suffix from examples
>      >      >      - Verified that -device virtio-gpu-rutabaga works. 
>     Strangely
>      >      >        enough, I don't remember changing anything, and I
>     remember
>      >      >        it not working.  I did rebase to top of tree though.
>      >      >      - Fixed meson examples in crosvm docs
>      >      >
>      >      >   docs/system/device-emulation.rst   |   1 +
>      >      >   docs/system/devices/virtio-gpu.rst | 113
>      >     +++++++++++++++++++++++++++++
>      >      >   2 files changed, 114 insertions(+)
>      >      >   create mode 100644 docs/system/devices/virtio-gpu.rst
>      >      >
>      >      > diff --git a/docs/system/device-emulation.rst
>      >     b/docs/system/device-emulation.rst
>      >      > index 4491c4cbf7..1167f3a9f2 100644
>      >      > --- a/docs/system/device-emulation.rst
>      >      > +++ b/docs/system/device-emulation.rst
>      >      > @@ -91,6 +91,7 @@ Emulated Devices
>      >      >      devices/nvme.rst
>      >      >      devices/usb.rst
>      >      >      devices/vhost-user.rst
>      >      > +   devices/virtio-gpu.rst
>      >      >      devices/virtio-pmem.rst
>      >      >      devices/vhost-user-rng.rst
>      >      >      devices/canokey.rst
>      >      > diff --git a/docs/system/devices/virtio-gpu.rst
>      >     b/docs/system/devices/virtio-gpu.rst
>      >      > new file mode 100644
>      >      > index 0000000000..8c5c708272
>      >      > --- /dev/null
>      >      > +++ b/docs/system/devices/virtio-gpu.rst
>      >      > @@ -0,0 +1,113 @@
>      >      > +..
>      >      > +   SPDX-License-Identifier: GPL-2.0
>      >      > +
>      >      > +virtio-gpu
>      >      > +==========
>      >      > +
>      >      > +This document explains the setup and usage of the
>     virtio-gpu device.
>      >      > +The virtio-gpu device paravirtualizes the GPU and display
>      >     controller.
>      >      > +
>      >      > +Linux kernel support
>      >      > +--------------------
>      >      > +
>      >      > +virtio-gpu requires a guest Linux kernel built with the
>      >      > +``CONFIG_DRM_VIRTIO_GPU`` option.
>      >      > +
>      >      > +QEMU virtio-gpu variants
>      >      > +------------------------
>      >      > +
>      >      > +QEMU virtio-gpu device variants come in the following form:
>      >      > +
>      >      > + * ``virtio-vga[-BACKEND]``
>      >      > + * ``virtio-gpu[-BACKEND][-INTERFACE]``
>      >      > + * ``vhost-user-vga``
>      >      > + * ``vhost-user-pci``
>      >      > +
>      >      > +**Backends:** QEMU provides a 2D virtio-gpu backend, and two
>      >     accelerated
>      >      > +backends: virglrenderer ('gl' device label) and rutabaga_gfx
>      >     ('rutabaga'
>      >      > +device label).  There is a vhost-user backend that runs the
>      >     graphics stack
>      >      > +in a separate process for improved isolation.
>      >      > +
>      >      > +**Interfaces:** QEMU further categorizes virtio-gpu device
>      >     variants based
>      >      > +on the interface exposed to the guest. The interfaces can be
>      >     classified
>      >      > +into VGA and non-VGA variants. The VGA ones are prefixed with
>      >     virtio-vga
>      >      > +or vhost-user-vga while the non-VGA ones are prefixed with
>      >     virtio-gpu or
>      >      > +vhost-user-gpu.
>      >      > +
>      >      > +The VGA ones always use the PCI interface, but for the
>     non-VGA
>      >     ones, the
>      >      > +user can further pick between MMIO or PCI. For MMIO, the user
>      >     can suffix
>      >      > +the device name with -device, though vhost-user-gpu does not
>      >     support MMIO.
>      >      > +For PCI, the user can suffix it with -pci. Without these
>      >     suffixes, the
>      >      > +platform default will be chosen.
>      >      > +
>      >      > +virtio-gpu 2d
>      >      > +-------------
>      >      > +
>      >      > +The default 2D backend only performs 2D operations. The guest
>      >     needs to
>      >      > +employ a software renderer for 3D graphics.
>      >      > +
>      >      > +Typically, the software renderer is provided by `Mesa`_ or
>      >     `SwiftShader`_.
>      >      > +Mesa's implementations (LLVMpipe, Lavapipe and virgl
>     below) work
>      >     out of box
>      >      > +on typical modern Linux distributions.
>      >      > +
>      >      > +.. parsed-literal::
>      >      > +    -device virtio-gpu
>      >      > +
>      >      > +.. _Mesa: https://www.mesa3d.org/
>     <https://www.mesa3d.org/> <https://www.mesa3d.org/
>     <https://www.mesa3d.org/>>
>      >      > +.. _SwiftShader: https://github.com/google/swiftshader
>     <https://github.com/google/swiftshader>
>      >     <https://github.com/google/swiftshader
>     <https://github.com/google/swiftshader>>
>      >      > +
>      >      > +virtio-gpu virglrenderer
>      >      > +------------------------
>      >      > +
>      >      > +When using virgl accelerated graphics mode in the guest,
>     OpenGL
>      >     API calls
>      >      > +are translated into an intermediate representation (see
>      >     `Gallium3D`_). The
>      >      > +intermediate representation is communicated to the host
>     and the
>      >      > +`virglrenderer`_ library on the host translates the
>     intermediate
>      >      > +representation back to OpenGL API calls.
>      >      > +
>      >      > +.. parsed-literal::
>      >      > +    -device virtio-gpu-gl
>      >      > +
>      >      > +.. _Gallium3D:
>      > https://www.freedesktop.org/wiki/Software/gallium/
>     <https://www.freedesktop.org/wiki/Software/gallium/>
>      >     <https://www.freedesktop.org/wiki/Software/gallium/
>     <https://www.freedesktop.org/wiki/Software/gallium/>>
>      >      > +.. _virglrenderer:
>      > https://gitlab.freedesktop.org/virgl/virglrenderer/
>     <https://gitlab.freedesktop.org/virgl/virglrenderer/>
>      >     <https://gitlab.freedesktop.org/virgl/virglrenderer/
>     <https://gitlab.freedesktop.org/virgl/virglrenderer/>>
>      >      > +
>      >      > +virtio-gpu rutabaga
>      >      > +-------------------
>      >      > +
>      >      > +virtio-gpu can also leverage `rutabaga_gfx`_ to provide
>     `gfxstream`_
>      >      > +rendering and `Wayland display passthrough`_.  With the
>      >     gfxstream rendering
>      >      > +mode, GLES and Vulkan calls are forwarded to the host
>     with minimal
>      >      > +modification.
>      >      > +
>      >      > +The crosvm book provides directions on how to build a
>      >     `gfxstream-enabled
>      >      > +rutabaga`_ and launch a `guest Wayland proxy`_.
>      >      > +
>      >      > +This device does require host blob support (``hostmem`` field
>      >     below). The
>      >      > +``hostmem`` field specifies the size of virtio-gpu host
>     memory
>      >     window.
>      >      > +This is typically between 256M and 8G.
>      >      > +
>      >      > +At least one capset (see colon separated ``capset_names``
>     below)
>      >     must be
>      >      > +specified when starting the device.  The currently supported
>      >      > +``capset_names`` are ``gfxstream-vulkan`` and
>     ``cross-domain``
>      >     on Linux
>      >      > +guests. For Android guests, ``gfxstream-gles`` is also
>     supported.
>      >      > +
>      >      > +The device will try to auto-detect the wayland socket
>     path if the
>      >      > +``cross-domain`` capset name is set.  The user may optionally
>      >     specify
>      >      > +``wayland_socket_path`` for non-standard paths.
>      >      > +
>      >      > +The ``wsi`` option can be set to ``surfaceless`` or
>     ``headless``.
>      >      > +Surfaceless doesn't create a native window surface, but does
>      >     copy from the
>      >      > +render target to the Pixman buffer if a virtio-gpu 2D
>     hypercall
>      >     is issued.
>      >      > +Headless is like surfaceless, but doesn't copy to the
>     Pixman buffer.
>      >      > +Surfaceless is the default if ``wsi`` is not specified.
>      >      > +
>      >      > +.. parsed-literal::
>      >      > +    -device
>      >     virtio-gpu-rutabaga,capset_names=gfxstream-vulkan:cross-domain,
>      >      > +
>      >     
>       hostmem=8G,wayland_socket_path=/tmp/nonstandard/mock_wayland.sock,
>      >      > +       wsi=headless
>      >      > +
>      >      > +.. _rutabaga_gfx:
>      >
>     https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h <https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h> <https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h <https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h>>
>      >      > +.. _gfxstream:
>      >
>     https://android.googlesource.com/platform/hardware/google/gfxstream/
>     <https://android.googlesource.com/platform/hardware/google/gfxstream/>
>      >   
>       <https://android.googlesource.com/platform/hardware/google/gfxstream/ <https://android.googlesource.com/platform/hardware/google/gfxstream/>>
>      >      > +.. _Wayland display passthrough:
>      > https://www.youtube.com/watch?v=OZJiHMtIQ2M
>     <https://www.youtube.com/watch?v=OZJiHMtIQ2M>
>      >     <https://www.youtube.com/watch?v=OZJiHMtIQ2M
>     <https://www.youtube.com/watch?v=OZJiHMtIQ2M>>
>      >      > +.. _gfxstream-enabled rutabaga:
>      > https://crosvm.dev/book/appendix/rutabaga_gfx.html
>     <https://crosvm.dev/book/appendix/rutabaga_gfx.html>
>      >     <https://crosvm.dev/book/appendix/rutabaga_gfx.html
>     <https://crosvm.dev/book/appendix/rutabaga_gfx.html>>
>      >
>      >     You have different links for "rutabaga_gfx" and
>     "gfxstream-enabled
>      >     rutabaga", but I think you only need one.
>      >
>      >
>      > Done.  Didn't resend the entire patch series (to avoid spamming
>     list),
>      > just did "in-reply-to".  The change is also available at:
>      >
>      >
>     https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8 <https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8> <https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8 <https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8>>
> 
>     The patch series now looks good so I finally decided to try this patch
>     series, but I couldn't get it work.
> 
>     I noticed gfxstream has page size hardcoded as 4 KiB, which broke my
>     setup on M2 MacBook Air (running Asahi Linux) that has 16 KiB page. You
>     may add code to check host page size and to report an error if it's not
>     4 KiB to virtio-gpu-rutabaga, but I think it's trivial to fix gfxstream
>     to query page size at runtime as QEMU and Rutabaga does so I hope
>     you to
>     do so. For testing purpose, I have replaced it with 16 KiB on my setup.
> 
> 
> Good catch, the fix to the 16KiB was merged today and is in gfxstream 
> ToT right now.

The fix is incomplete. There are a few other places that hardcodes page 
size, namely ANDROID_EMU_ADDRESS_SPACE_DEFAULT_PAGE_SIZE and 
ADDRESS_SPACE_GRAPHICS_PAGE_SIZE.

ANDROID_EMU_ADDRESS_SPACE_DEFAULT_PAGE_SIZE is used by no one so you can 
just remove it. ADDRESS_SPACE_GRAPHICS_PAGE_SIZE is actually in use and 
needs to be fixed.

It's also better to remove PAGE_SIZE definition from guest/meson.build 
just in case.

> 
> 
>     I also found some bugs on QEMU side so I added comments to respective
>     patches.
> 
>     Below is the logs from my last attempt of running vkgears:
> 
>     $ VK_LOADER_DEBUG=all demos/build/src/vulkan/vkgears
>     INFO:             Vulkan Loader Version 1.3.243
>     LAYER:            Searching for layer manifest files
>     LAYER:               In following locations:
>     LAYER:                  /var/home/person/.config/vulkan/implicit_layer.d
>     LAYER:                  /etc/xdg/vulkan/implicit_layer.d
>     LAYER:                  /etc/vulkan/implicit_layer.d
>     LAYER:
>     /var/home/person/.local/share/vulkan/implicit_layer.d
>     LAYER:
>     /var/home/person/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
>     LAYER:
>     /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
>     LAYER:                  /usr/local/share/vulkan/implicit_layer.d
>     LAYER:                  /usr/share/vulkan/implicit_layer.d
>     LAYER:               Found the following files:
>     LAYER:
>     /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
>     INFO:             Found manifest file
>     /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
>     (file
>     version "1.0.0")
>     LAYER:            Searching for layer manifest files
>     LAYER:               In following locations:
>     LAYER:                  /var/home/person/.config/vulkan/explicit_layer.d
>     LAYER:                  /etc/xdg/vulkan/explicit_layer.d
>     LAYER:                  /etc/vulkan/explicit_layer.d
>     LAYER:
>     /var/home/person/.local/share/vulkan/explicit_layer.d
>     LAYER:
>     /var/home/person/.local/share/flatpak/exports/share/vulkan/explicit_layer.d
>     LAYER:
>     /var/lib/flatpak/exports/share/vulkan/explicit_layer.d
>     LAYER:                  /usr/local/share/vulkan/explicit_layer.d
>     LAYER:                  /usr/share/vulkan/explicit_layer.d
>     LAYER:               Found no files
>     DRIVER:           Searching for driver manifest files
>     DRIVER:              In following locations:
>     DRIVER:                 /var/home/person/.config/vulkan/icd.d
>     DRIVER:                 /etc/xdg/vulkan/icd.d
>     DRIVER:                 /etc/vulkan/icd.d
>     DRIVER:                 /var/home/person/.local/share/vulkan/icd.d
>     DRIVER:
>     /var/home/person/.local/share/flatpak/exports/share/vulkan/icd.d
>     DRIVER:                 /var/lib/flatpak/exports/share/vulkan/icd.d
>     DRIVER:                 /usr/local/share/vulkan/icd.d
>     DRIVER:                 /usr/share/vulkan/icd.d
>     DRIVER:              Found the following files:
>     DRIVER:
>     /usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json
>     DRIVER:               
>       /usr/share/vulkan/icd.d/broadcom_icd.aarch64.json
>     DRIVER:               
>       /usr/share/vulkan/icd.d/freedreno_icd.aarch64.json
>     DRIVER:                 /usr/share/vulkan/icd.d/lvp_icd.aarch64.json
>     DRIVER:               
>       /usr/share/vulkan/icd.d/panfrost_icd.aarch64.json
>     DRIVER:                 /usr/share/vulkan/icd.d/radeon_icd.aarch64.json
>     DRIVER:           Found ICD manifest file
>     /usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json, version "1.0.0"
>     DEBUG | DRIVER:   Searching for ICD drivers named
>     /usr/local/lib64/libvulkan_cereal.so
>     [VirtGpuDevice.cpp(71)]
>     [prio 6] virtgpu backend not enabling
>     VIRTGPU_PARAM_CREATE_GUEST_HANDLE[AndroidHealthMonitor.cpp(275)]
>     [prio 4] HealthMonitor disabled. Returning nullptrI0818 21:00:07.980257
>     154451 IntelDrmDecoder.cpp:38] IntelDrmDecoder created for context 2
>     DRIVER:           Found ICD manifest file
>     /usr/share/vulkan/icd.d/broadcom_icd.aarch64.json, version "1.0.0"
>     DEBUG | DRIVER:   Searching for ICD drivers named
>     /usr/lib64/libvulkan_broadcom.so
>     DRIVER:           Found ICD manifest file
>     /usr/share/vulkan/icd.d/freedreno_icd.aarch64.json, version "1.0.0"
>     DEBUG | DRIVER:   Searching for ICD drivers named
>     /usr/lib64/libvulkan_freedreno.so
>     DRIVER:           Found ICD manifest file
>     /usr/share/vulkan/icd.d/lvp_icd.aarch64.json, version "1.0.0"
>     DEBUG | DRIVER:   Searching for ICD drivers named
>     /usr/lib64/libvulkan_lvp.so
>     DRIVER:           Found ICD manifest file
>     /usr/share/vulkan/icd.d/panfrost_icd.aarch64.json, version "1.0.0"
>     DEBUG | DRIVER:   Searching for ICD drivers named
>     /usr/lib64/libvulkan_panfrost.so
>     DRIVER:           Found ICD manifest file
>     /usr/share/vulkan/icd.d/radeon_icd.aarch64.json, version "1.0.0"
>     DEBUG | DRIVER:   Searching for ICD drivers named
>     /usr/lib64/libvulkan_radeon.so
>     DEBUG | LAYER:    Loading layer library libVkLayer_MESA_device_select.so
>     INFO | LAYER:     Insert instance layer "VK_LAYER_MESA_device_select"
>     (libVkLayer_MESA_device_select.so)
>     LAYER:            vkCreateInstance layer callstack setup to:
>     LAYER:               <Application>
>     LAYER:                 ||
>     LAYER:               <Loader>
>     LAYER:                 ||
>     LAYER:               VK_LAYER_MESA_device_select
>     LAYER:                       Type: Implicit
>     LAYER:                           Disable Env Var:  NODEVICE_SELECT
>     LAYER:                       Manifest:
>     /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
>     LAYER:                       Library:  libVkLayer_MESA_device_select.so
>     LAYER:                 ||
>     LAYER:               <Drivers>
>     I0818 21:00:08.014896  154451 VkDecoderGlobalState.cpp:443] Creating
>     Vulkan instance for app: vkgears
>     INFO | DRIVER:    linux_read_sorted_physical_devices:
>     INFO | DRIVER:         Original order:
>     INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
>     INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
>     INFO | DRIVER:         Sorted order:
>     INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
>     INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
>     INFO | DRIVER:    linux_read_sorted_physical_devices:
>     INFO | DRIVER:         Original order:
>     INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
>     INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
>     INFO | DRIVER:         Sorted order:
>     INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
>     INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
>     DEBUG | DRIVER:   Copying old device 0 into new device 0
>     DEBUG | DRIVER:   Copying old device 1 into new device 1
>     INFO | DRIVER:    linux_read_sorted_physical_devices:
>     INFO | DRIVER:         Original order:
>     INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
>     INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
>     INFO | DRIVER:         Sorted order:
>     INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
>     INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
>     DEBUG | DRIVER:   Copying old device 0 into new device 0
>     DEBUG | DRIVER:   Copying old device 1 into new device 1
>     INFO | DRIVER:    linux_read_sorted_physical_devices:
>     INFO | DRIVER:         Original order:
>     INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
>     INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
>     INFO | DRIVER:         Sorted order:
>     INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
>     INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
>     DEBUG | DRIVER:   Copying old device 0 into new device 0
>     DEBUG | DRIVER:   Copying old device 1 into new device 1
>     ERROR:            loader_validate_device_extensions: Device extension
>     VK_KHR_swapchain not supported by selected physical device or enabled
> 
> 
> Yeah, any non-headless Linux tests are unlikely to work.  Maybe in a 
> future gfxstream release, since our focus is of course on Android and 
> getting Vulkan in QEMU in general.

I have just tried "vulkan-samples sample hello_triangle --headless" but 
no luck. Below is the output of QEMU with -trace virtio_gpu_cmd_*

virtio_gpu_cmd_ctx_create ctx 0x2, name vulkan_samples
virtio_gpu_cmd_res_create_blob res 0x24, size 1064960
virtio_gpu_cmd_ctx_res_attach ctx 0x2, res 0x24
virtio_gpu_cmd_ctx_submit ctx 0x2, size 8
I0819 15:10:06.916033   21850 IntelDrmDecoder.cpp:38] IntelDrmDecoder 
created for context 2
virtio_gpu_cmd_ctx_submit ctx 0x2, size 8
virtio_gpu_cmd_ctx_submit ctx 0x2, size 8
W0819 15:10:06.916443   21850 VkDecoder.cpp:137] Bad packet length 0 
detected, decode may fail
virtio_gpu_cmd_ctx_submit ctx 0x2, size 8
W0819 15:10:06.916465   21850 VkDecoder.cpp:137] Bad packet length 0 
detected, decode may fail
W0819 15:10:06.916473   21850 VkDecoder.cpp:137] Bad packet length 0 
detected, decode may fail
W0819 15:10:06.916478   21850 VkDecoder.cpp:137] Bad packet length 0 
detected, decode may fail
W0819 15:10:06.916482   21850 VkDecoder.cpp:137] Bad packet length 0 
detected, decode may fail

And it endlessly outputs "Bad packet length" error messages.

Gurchetan Singh Aug. 22, 2023, 12:20 a.m. UTC | #6

On Fri, Aug 18, 2023 at 11:13 PM Akihiko Odaki <[e-mail]>
wrote:

> On 2023/08/19 10:17, Gurchetan Singh wrote:
> >
> >
> > On Fri, Aug 18, 2023 at 5:08 AM Akihiko Odaki <[e-mail]
> > <mailto:[e-mail]>> wrote:
> >
> >     On 2023/08/18 8:47, Gurchetan Singh wrote:
> >      >
> >      >
> >      > On Wed, Aug 16, 2023 at 10:28 PM Akihiko Odaki
> >     <[e-mail] <mailto:[e-mail]>
> >      > <mailto:[e-mail]
> >     <mailto:[e-mail]>>> wrote:
> >      >
> >      >     On 2023/08/17 11:23, Gurchetan Singh wrote:
> >      >      > From: Gurchetan Singh <[e-mail]
> >     <mailto:[e-mail]>
> >      >     <mailto:[e-mail]
> >     <mailto:[e-mail]>>>
> >      >      >
> >      >      > This adds basic documentation for virtio-gpu.
> >      >      >
> >      >      > Suggested-by: Akihiko Odaki <[e-mail]
> >     <mailto:[e-mail]>
> >      >     <mailto:[e-mail]
> >     <mailto:[e-mail]>>>
> >      >      > Signed-off-by: Gurchetan Singh
> >     <[e-mail] <mailto:[e-mail]>
> >      >     <mailto:[e-mail]
> >     <mailto:[e-mail]>>>
> >      >      > Tested-by: Alyssa Ross <[e-mail] <mailto:[e-mail]>
> >     <mailto:[e-mail] <mailto:[e-mail]>>>
> >      >      > Tested-by: Emmanouil Pitsidianakis
> >      >     <[e-mail]
> >     <mailto:[e-mail]>
> >     <mailto:[e-mail]
> >     <mailto:[e-mail]>>>
> >      >      > Reviewed-by: Emmanouil Pitsidianakis
> >      >     <[e-mail]
> >     <mailto:[e-mail]>
> >     <mailto:[e-mail]
> >     <mailto:[e-mail]>>>
> >      >      > ---
> >      >      > v2: - Incorporated suggestions by Akihiko Odaki
> >      >      >      - Listed the currently supported capset_names
> (Bernard)
> >      >      >
> >      >      > v3: - Incorporated suggestions by Akihiko Odaki and Alyssa
> >     Ross
> >      >      >
> >      >      > v4: - Incorporated suggestions by Akihiko Odaki
> >      >      >
> >      >      > v5: - Removed pci suffix from examples
> >      >      >      - Verified that -device virtio-gpu-rutabaga works.
> >     Strangely
> >      >      >        enough, I don't remember changing anything, and I
> >     remember
> >      >      >        it not working.  I did rebase to top of tree though.
> >      >      >      - Fixed meson examples in crosvm docs
> >      >      >
> >      >      >   docs/system/device-emulation.rst   |   1 +
> >      >      >   docs/system/devices/virtio-gpu.rst | 113
> >      >     +++++++++++++++++++++++++++++
> >      >      >   2 files changed, 114 insertions(+)
> >      >      >   create mode 100644 docs/system/devices/virtio-gpu.rst
> >      >      >
> >      >      > diff --git a/docs/system/device-emulation.rst
> >      >     b/docs/system/device-emulation.rst
> >      >      > index 4491c4cbf7..1167f3a9f2 100644
> >      >      > --- a/docs/system/device-emulation.rst
> >      >      > +++ b/docs/system/device-emulation.rst
> >      >      > @@ -91,6 +91,7 @@ Emulated Devices
> >      >      >      devices/nvme.rst
> >      >      >      devices/usb.rst
> >      >      >      devices/vhost-user.rst
> >      >      > +   devices/virtio-gpu.rst
> >      >      >      devices/virtio-pmem.rst
> >      >      >      devices/vhost-user-rng.rst
> >      >      >      devices/canokey.rst
> >      >      > diff --git a/docs/system/devices/virtio-gpu.rst
> >      >     b/docs/system/devices/virtio-gpu.rst
> >      >      > new file mode 100644
> >      >      > index 0000000000..8c5c708272
> >      >      > --- /dev/null
> >      >      > +++ b/docs/system/devices/virtio-gpu.rst
> >      >      > @@ -0,0 +1,113 @@
> >      >      > +..
> >      >      > +   SPDX-License-Identifier: GPL-2.0
> >      >      > +
> >      >      > +virtio-gpu
> >      >      > +==========
> >      >      > +
> >      >      > +This document explains the setup and usage of the
> >     virtio-gpu device.
> >      >      > +The virtio-gpu device paravirtualizes the GPU and display
> >      >     controller.
> >      >      > +
> >      >      > +Linux kernel support
> >      >      > +--------------------
> >      >      > +
> >      >      > +virtio-gpu requires a guest Linux kernel built with the
> >      >      > +``CONFIG_DRM_VIRTIO_GPU`` option.
> >      >      > +
> >      >      > +QEMU virtio-gpu variants
> >      >      > +------------------------
> >      >      > +
> >      >      > +QEMU virtio-gpu device variants come in the following
> form:
> >      >      > +
> >      >      > + * ``virtio-vga[-BACKEND]``
> >      >      > + * ``virtio-gpu[-BACKEND][-INTERFACE]``
> >      >      > + * ``vhost-user-vga``
> >      >      > + * ``vhost-user-pci``
> >      >      > +
> >      >      > +**Backends:** QEMU provides a 2D virtio-gpu backend, and
> two
> >      >     accelerated
> >      >      > +backends: virglrenderer ('gl' device label) and
> rutabaga_gfx
> >      >     ('rutabaga'
> >      >      > +device label).  There is a vhost-user backend that runs
> the
> >      >     graphics stack
> >      >      > +in a separate process for improved isolation.
> >      >      > +
> >      >      > +**Interfaces:** QEMU further categorizes virtio-gpu device
> >      >     variants based
> >      >      > +on the interface exposed to the guest. The interfaces can
> be
> >      >     classified
> >      >      > +into VGA and non-VGA variants. The VGA ones are prefixed
> with
> >      >     virtio-vga
> >      >      > +or vhost-user-vga while the non-VGA ones are prefixed with
> >      >     virtio-gpu or
> >      >      > +vhost-user-gpu.
> >      >      > +
> >      >      > +The VGA ones always use the PCI interface, but for the
> >     non-VGA
> >      >     ones, the
> >      >      > +user can further pick between MMIO or PCI. For MMIO, the
> user
> >      >     can suffix
> >      >      > +the device name with -device, though vhost-user-gpu does
> not
> >      >     support MMIO.
> >      >      > +For PCI, the user can suffix it with -pci. Without these
> >      >     suffixes, the
> >      >      > +platform default will be chosen.
> >      >      > +
> >      >      > +virtio-gpu 2d
> >      >      > +-------------
> >      >      > +
> >      >      > +The default 2D backend only performs 2D operations. The
> guest
> >      >     needs to
> >      >      > +employ a software renderer for 3D graphics.
> >      >      > +
> >      >      > +Typically, the software renderer is provided by `Mesa`_ or
> >      >     `SwiftShader`_.
> >      >      > +Mesa's implementations (LLVMpipe, Lavapipe and virgl
> >     below) work
> >      >     out of box
> >      >      > +on typical modern Linux distributions.
> >      >      > +
> >      >      > +.. parsed-literal::
> >      >      > +    -device virtio-gpu
> >      >      > +
> >      >      > +.. _Mesa: https://www.mesa3d.org/
> >     <https://www.mesa3d.org/> <https://www.mesa3d.org/
> >     <https://www.mesa3d.org/>>
> >      >      > +.. _SwiftShader: https://github.com/google/swiftshader
> >     <https://github.com/google/swiftshader>
> >      >     <https://github.com/google/swiftshader
> >     <https://github.com/google/swiftshader>>
> >      >      > +
> >      >      > +virtio-gpu virglrenderer
> >      >      > +------------------------
> >      >      > +
> >      >      > +When using virgl accelerated graphics mode in the guest,
> >     OpenGL
> >      >     API calls
> >      >      > +are translated into an intermediate representation (see
> >      >     `Gallium3D`_). The
> >      >      > +intermediate representation is communicated to the host
> >     and the
> >      >      > +`virglrenderer`_ library on the host translates the
> >     intermediate
> >      >      > +representation back to OpenGL API calls.
> >      >      > +
> >      >      > +.. parsed-literal::
> >      >      > +    -device virtio-gpu-gl
> >      >      > +
> >      >      > +.. _Gallium3D:
> >      > https://www.freedesktop.org/wiki/Software/gallium/
> >     <https://www.freedesktop.org/wiki/Software/gallium/>
> >      >     <https://www.freedesktop.org/wiki/Software/gallium/
> >     <https://www.freedesktop.org/wiki/Software/gallium/>>
> >      >      > +.. _virglrenderer:
> >      > https://gitlab.freedesktop.org/virgl/virglrenderer/
> >     <https://gitlab.freedesktop.org/virgl/virglrenderer/>
> >      >     <https://gitlab.freedesktop.org/virgl/virglrenderer/
> >     <https://gitlab.freedesktop.org/virgl/virglrenderer/>>
> >      >      > +
> >      >      > +virtio-gpu rutabaga
> >      >      > +-------------------
> >      >      > +
> >      >      > +virtio-gpu can also leverage `rutabaga_gfx`_ to provide
> >     `gfxstream`_
> >      >      > +rendering and `Wayland display passthrough`_.  With the
> >      >     gfxstream rendering
> >      >      > +mode, GLES and Vulkan calls are forwarded to the host
> >     with minimal
> >      >      > +modification.
> >      >      > +
> >      >      > +The crosvm book provides directions on how to build a
> >      >     `gfxstream-enabled
> >      >      > +rutabaga`_ and launch a `guest Wayland proxy`_.
> >      >      > +
> >      >      > +This device does require host blob support (``hostmem``
> field
> >      >     below). The
> >      >      > +``hostmem`` field specifies the size of virtio-gpu host
> >     memory
> >      >     window.
> >      >      > +This is typically between 256M and 8G.
> >      >      > +
> >      >      > +At least one capset (see colon separated ``capset_names``
> >     below)
> >      >     must be
> >      >      > +specified when starting the device.  The currently
> supported
> >      >      > +``capset_names`` are ``gfxstream-vulkan`` and
> >     ``cross-domain``
> >      >     on Linux
> >      >      > +guests. For Android guests, ``gfxstream-gles`` is also
> >     supported.
> >      >      > +
> >      >      > +The device will try to auto-detect the wayland socket
> >     path if the
> >      >      > +``cross-domain`` capset name is set.  The user may
> optionally
> >      >     specify
> >      >      > +``wayland_socket_path`` for non-standard paths.
> >      >      > +
> >      >      > +The ``wsi`` option can be set to ``surfaceless`` or
> >     ``headless``.
> >      >      > +Surfaceless doesn't create a native window surface, but
> does
> >      >     copy from the
> >      >      > +render target to the Pixman buffer if a virtio-gpu 2D
> >     hypercall
> >      >     is issued.
> >      >      > +Headless is like surfaceless, but doesn't copy to the
> >     Pixman buffer.
> >      >      > +Surfaceless is the default if ``wsi`` is not specified.
> >      >      > +
> >      >      > +.. parsed-literal::
> >      >      > +    -device
> >      >
>  virtio-gpu-rutabaga,capset_names=gfxstream-vulkan:cross-domain,
> >      >      > +
> >      >
> >       hostmem=8G,wayland_socket_path=/tmp/nonstandard/mock_wayland.sock,
> >      >      > +       wsi=headless
> >      >      > +
> >      >      > +.. _rutabaga_gfx:
> >      >
> >
> https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h
> <
> https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h>
> <
> https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h
> <
> https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h
> >>
> >      >      > +.. _gfxstream:
> >      >
> >     https://android.googlesource.com/platform/hardware/google/gfxstream/
> >     <
> https://android.googlesource.com/platform/hardware/google/gfxstream/>
> >      >
> >       <
> https://android.googlesource.com/platform/hardware/google/gfxstream/ <
> https://android.googlesource.com/platform/hardware/google/gfxstream/>>
> >      >      > +.. _Wayland display passthrough:
> >      > https://www.youtube.com/watch?v=OZJiHMtIQ2M
> >     <https://www.youtube.com/watch?v=OZJiHMtIQ2M>
> >      >     <https://www.youtube.com/watch?v=OZJiHMtIQ2M
> >     <https://www.youtube.com/watch?v=OZJiHMtIQ2M>>
> >      >      > +.. _gfxstream-enabled rutabaga:
> >      > https://crosvm.dev/book/appendix/rutabaga_gfx.html
> >     <https://crosvm.dev/book/appendix/rutabaga_gfx.html>
> >      >     <https://crosvm.dev/book/appendix/rutabaga_gfx.html
> >     <https://crosvm.dev/book/appendix/rutabaga_gfx.html>>
> >      >
> >      >     You have different links for "rutabaga_gfx" and
> >     "gfxstream-enabled
> >      >     rutabaga", but I think you only need one.
> >      >
> >      >
> >      > Done.  Didn't resend the entire patch series (to avoid spamming
> >     list),
> >      > just did "in-reply-to".  The change is also available at:
> >      >
> >      >
> >
> https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8
> <
> https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8>
> <
> https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8
> <
> https://gitlab.freedesktop.org/gurchetansingh/qemu-gfxstream/-/commits/qemu-gfxstream-v8
> >>
> >
> >     The patch series now looks good so I finally decided to try this
> patch
> >     series, but I couldn't get it work.
> >
> >     I noticed gfxstream has page size hardcoded as 4 KiB, which broke my
> >     setup on M2 MacBook Air (running Asahi Linux) that has 16 KiB page.
> You
> >     may add code to check host page size and to report an error if it's
> not
> >     4 KiB to virtio-gpu-rutabaga, but I think it's trivial to fix
> gfxstream
> >     to query page size at runtime as QEMU and Rutabaga does so I hope
> >     you to
> >     do so. For testing purpose, I have replaced it with 16 KiB on my
> setup.
> >
> >
> > Good catch, the fix to the 16KiB was merged today and is in gfxstream
> > ToT right now.
>
> The fix is incomplete. There are a few other places that hardcodes page
> size, namely ANDROID_EMU_ADDRESS_SPACE_DEFAULT_PAGE_SIZE and
> ADDRESS_SPACE_GRAPHICS_PAGE_SIZE.
>
> ANDROID_EMU_ADDRESS_SPACE_DEFAULT_PAGE_SIZE is used by no one so you can
> just remove it. ADDRESS_SPACE_GRAPHICS_PAGE_SIZE is actually in use and
> needs to be fixed.
>
> It's also better to remove PAGE_SIZE definition from guest/meson.build
> just in case.
>

ANDROID_EMU_ADDRESS_SPACE_DEFAULT_PAGE_SIZE and  PAGE_SIZE were removed
from gfxstream ToT.
The host alignment of the ring blob is also now based on getpagesize(..)
and not hardcoded.


> >
> >
> >     I also found some bugs on QEMU side so I added comments to respective
> >     patches.
> >
> >     Below is the logs from my last attempt of running vkgears:
> >
> >     $ VK_LOADER_DEBUG=all demos/build/src/vulkan/vkgears
> >     INFO:             Vulkan Loader Version 1.3.243
> >     LAYER:            Searching for layer manifest files
> >     LAYER:               In following locations:
> >     LAYER:
> /var/home/person/.config/vulkan/implicit_layer.d
> >     LAYER:                  /etc/xdg/vulkan/implicit_layer.d
> >     LAYER:                  /etc/vulkan/implicit_layer.d
> >     LAYER:
> >     /var/home/person/.local/share/vulkan/implicit_layer.d
> >     LAYER:
> >
>  /var/home/person/.local/share/flatpak/exports/share/vulkan/implicit_layer.d
> >     LAYER:
> >     /var/lib/flatpak/exports/share/vulkan/implicit_layer.d
> >     LAYER:                  /usr/local/share/vulkan/implicit_layer.d
> >     LAYER:                  /usr/share/vulkan/implicit_layer.d
> >     LAYER:               Found the following files:
> >     LAYER:
> >     /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
> >     INFO:             Found manifest file
> >     /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
> >     (file
> >     version "1.0.0")
> >     LAYER:            Searching for layer manifest files
> >     LAYER:               In following locations:
> >     LAYER:
> /var/home/person/.config/vulkan/explicit_layer.d
> >     LAYER:                  /etc/xdg/vulkan/explicit_layer.d
> >     LAYER:                  /etc/vulkan/explicit_layer.d
> >     LAYER:
> >     /var/home/person/.local/share/vulkan/explicit_layer.d
> >     LAYER:
> >
>  /var/home/person/.local/share/flatpak/exports/share/vulkan/explicit_layer.d
> >     LAYER:
> >     /var/lib/flatpak/exports/share/vulkan/explicit_layer.d
> >     LAYER:                  /usr/local/share/vulkan/explicit_layer.d
> >     LAYER:                  /usr/share/vulkan/explicit_layer.d
> >     LAYER:               Found no files
> >     DRIVER:           Searching for driver manifest files
> >     DRIVER:              In following locations:
> >     DRIVER:                 /var/home/person/.config/vulkan/icd.d
> >     DRIVER:                 /etc/xdg/vulkan/icd.d
> >     DRIVER:                 /etc/vulkan/icd.d
> >     DRIVER:                 /var/home/person/.local/share/vulkan/icd.d
> >     DRIVER:
> >     /var/home/person/.local/share/flatpak/exports/share/vulkan/icd.d
> >     DRIVER:                 /var/lib/flatpak/exports/share/vulkan/icd.d
> >     DRIVER:                 /usr/local/share/vulkan/icd.d
> >     DRIVER:                 /usr/share/vulkan/icd.d
> >     DRIVER:              Found the following files:
> >     DRIVER:
> >     /usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json
> >     DRIVER:
> >       /usr/share/vulkan/icd.d/broadcom_icd.aarch64.json
> >     DRIVER:
> >       /usr/share/vulkan/icd.d/freedreno_icd.aarch64.json
> >     DRIVER:                 /usr/share/vulkan/icd.d/lvp_icd.aarch64.json
> >     DRIVER:
> >       /usr/share/vulkan/icd.d/panfrost_icd.aarch64.json
> >     DRIVER:
>  /usr/share/vulkan/icd.d/radeon_icd.aarch64.json
> >     DRIVER:           Found ICD manifest file
> >     /usr/local/share/vulkan/icd.d/cereal_icd.aarch64.json, version
> "1.0.0"
> >     DEBUG | DRIVER:   Searching for ICD drivers named
> >     /usr/local/lib64/libvulkan_cereal.so
> >     [VirtGpuDevice.cpp(71)]
> >     [prio 6] virtgpu backend not enabling
> >     VIRTGPU_PARAM_CREATE_GUEST_HANDLE[AndroidHealthMonitor.cpp(275)]
> >     [prio 4] HealthMonitor disabled. Returning nullptrI0818
> 21:00:07.980257
> >     154451 IntelDrmDecoder.cpp:38] IntelDrmDecoder created for context 2
> >     DRIVER:           Found ICD manifest file
> >     /usr/share/vulkan/icd.d/broadcom_icd.aarch64.json, version "1.0.0"
> >     DEBUG | DRIVER:   Searching for ICD drivers named
> >     /usr/lib64/libvulkan_broadcom.so
> >     DRIVER:           Found ICD manifest file
> >     /usr/share/vulkan/icd.d/freedreno_icd.aarch64.json, version "1.0.0"
> >     DEBUG | DRIVER:   Searching for ICD drivers named
> >     /usr/lib64/libvulkan_freedreno.so
> >     DRIVER:           Found ICD manifest file
> >     /usr/share/vulkan/icd.d/lvp_icd.aarch64.json, version "1.0.0"
> >     DEBUG | DRIVER:   Searching for ICD drivers named
> >     /usr/lib64/libvulkan_lvp.so
> >     DRIVER:           Found ICD manifest file
> >     /usr/share/vulkan/icd.d/panfrost_icd.aarch64.json, version "1.0.0"
> >     DEBUG | DRIVER:   Searching for ICD drivers named
> >     /usr/lib64/libvulkan_panfrost.so
> >     DRIVER:           Found ICD manifest file
> >     /usr/share/vulkan/icd.d/radeon_icd.aarch64.json, version "1.0.0"
> >     DEBUG | DRIVER:   Searching for ICD drivers named
> >     /usr/lib64/libvulkan_radeon.so
> >     DEBUG | LAYER:    Loading layer library
> libVkLayer_MESA_device_select.so
> >     INFO | LAYER:     Insert instance layer "VK_LAYER_MESA_device_select"
> >     (libVkLayer_MESA_device_select.so)
> >     LAYER:            vkCreateInstance layer callstack setup to:
> >     LAYER:               <Application>
> >     LAYER:                 ||
> >     LAYER:               <Loader>
> >     LAYER:                 ||
> >     LAYER:               VK_LAYER_MESA_device_select
> >     LAYER:                       Type: Implicit
> >     LAYER:                           Disable Env Var:  NODEVICE_SELECT
> >     LAYER:                       Manifest:
> >     /usr/share/vulkan/implicit_layer.d/VkLayer_MESA_device_select.json
> >     LAYER:                       Library:
> libVkLayer_MESA_device_select.so
> >     LAYER:                 ||
> >     LAYER:               <Drivers>
> >     I0818 21:00:08.014896  154451 VkDecoderGlobalState.cpp:443] Creating
> >     Vulkan instance for app: vkgears
> >     INFO | DRIVER:    linux_read_sorted_physical_devices:
> >     INFO | DRIVER:         Original order:
> >     INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
> >     INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
> >     INFO | DRIVER:         Sorted order:
> >     INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
> >     INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
> >     INFO | DRIVER:    linux_read_sorted_physical_devices:
> >     INFO | DRIVER:         Original order:
> >     INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
> >     INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
> >     INFO | DRIVER:         Sorted order:
> >     INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
> >     INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
> >     DEBUG | DRIVER:   Copying old device 0 into new device 0
> >     DEBUG | DRIVER:   Copying old device 1 into new device 1
> >     INFO | DRIVER:    linux_read_sorted_physical_devices:
> >     INFO | DRIVER:         Original order:
> >     INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
> >     INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
> >     INFO | DRIVER:         Sorted order:
> >     INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
> >     INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
> >     DEBUG | DRIVER:   Copying old device 0 into new device 0
> >     DEBUG | DRIVER:   Copying old device 1 into new device 1
> >     INFO | DRIVER:    linux_read_sorted_physical_devices:
> >     INFO | DRIVER:         Original order:
> >     INFO | DRIVER:               [0] llvmpipe (LLVM 16.0.6, 128 bits)
> >     INFO | DRIVER:               [1] llvmpipe (LLVM 15.0.7, 128 bits)
> >     INFO | DRIVER:         Sorted order:
> >     INFO | DRIVER:               [0] llvmpipe (LLVM 15.0.7, 128 bits)
> >     INFO | DRIVER:               [1] llvmpipe (LLVM 16.0.6, 128 bits)
> >     DEBUG | DRIVER:   Copying old device 0 into new device 0
> >     DEBUG | DRIVER:   Copying old device 1 into new device 1
> >     ERROR:            loader_validate_device_extensions: Device extension
> >     VK_KHR_swapchain not supported by selected physical device or enabled
> >
> >
> > Yeah, any non-headless Linux tests are unlikely to work.  Maybe in a
> > future gfxstream release, since our focus is of course on Android and
> > getting Vulkan in QEMU in general.
>
> I have just tried "vulkan-samples sample hello_triangle --headless" but
> no luck. Below is the output of QEMU with -trace virtio_gpu_cmd_*
>

I have slightly different behavior on my setup, the app fails to find the
proper surface extensions and exits.   In headless mode, the app relies
on VK_EXT_headless_surface, which gfxstream does not currently support
(see on_vkEnumerateInstanceExtensionProperties).  For some reason, perhaps
due the presence of other VK drivers on the system, vulkaninfo does report
several X11/Wayland surface extensions with gfxstream.  This may confuse
some headless apps.

Either way, the crosvm docs will be clarified to mention what's been tested
regarding gfxstream guest Linux (crrev.com/c/4800334).  I suggest you run
deqp-vk if your goal is to test this patch series.


>
> virtio_gpu_cmd_ctx_create ctx 0x2, name vulkan_samples
> virtio_gpu_cmd_res_create_blob res 0x24, size 1064960
> virtio_gpu_cmd_ctx_res_attach ctx 0x2, res 0x24
> virtio_gpu_cmd_ctx_submit ctx 0x2, size 8
> I0819 15:10:06.916033   21850 IntelDrmDecoder.cpp:38] IntelDrmDecoder
> created for context 2
> virtio_gpu_cmd_ctx_submit ctx 0x2, size 8
> virtio_gpu_cmd_ctx_submit ctx 0x2, size 8
> W0819 15:10:06.916443   21850 VkDecoder.cpp:137] Bad packet length 0
> detected, decode may fail
> virtio_gpu_cmd_ctx_submit ctx 0x2, size 8
> W0819 15:10:06.916465   21850 VkDecoder.cpp:137] Bad packet length 0
> detected, decode may fail
> W0819 15:10:06.916473   21850 VkDecoder.cpp:137] Bad packet length 0
> detected, decode may fail
> W0819 15:10:06.916478   21850 VkDecoder.cpp:137] Bad packet length 0
> detected, decode may fail
> W0819 15:10:06.916482   21850 VkDecoder.cpp:137] Bad packet length 0
> detected, decode may fail
>
> And it endlessly outputs "Bad packet length" error messages.
>

Patch

diff --git a/docs/system/device-emulation.rst b/docs/system/device-emulation.rst
index 4491c4cbf7..1167f3a9f2 100644
--- a/docs/system/device-emulation.rst
+++ b/docs/system/device-emulation.rst
@@ -91,6 +91,7 @@  Emulated Devices
    devices/nvme.rst
    devices/usb.rst
    devices/vhost-user.rst
+   devices/virtio-gpu.rst
    devices/virtio-pmem.rst
    devices/vhost-user-rng.rst
    devices/canokey.rst
diff --git a/docs/system/devices/virtio-gpu.rst b/docs/system/devices/virtio-gpu.rst
new file mode 100644
index 0000000000..8c5c708272
--- /dev/null
+++ b/docs/system/devices/virtio-gpu.rst
@@ -0,0 +1,113 @@ 
+..
+   SPDX-License-Identifier: GPL-2.0
+
+virtio-gpu
+==========
+
+This document explains the setup and usage of the virtio-gpu device.
+The virtio-gpu device paravirtualizes the GPU and display controller.
+
+Linux kernel support
+--------------------
+
+virtio-gpu requires a guest Linux kernel built with the
+``CONFIG_DRM_VIRTIO_GPU`` option.
+
+QEMU virtio-gpu variants
+------------------------
+
+QEMU virtio-gpu device variants come in the following form:
+
+ * ``virtio-vga[-BACKEND]``
+ * ``virtio-gpu[-BACKEND][-INTERFACE]``
+ * ``vhost-user-vga``
+ * ``vhost-user-pci``
+
+**Backends:** QEMU provides a 2D virtio-gpu backend, and two accelerated
+backends: virglrenderer ('gl' device label) and rutabaga_gfx ('rutabaga'
+device label).  There is a vhost-user backend that runs the graphics stack
+in a separate process for improved isolation.
+
+**Interfaces:** QEMU further categorizes virtio-gpu device variants based
+on the interface exposed to the guest. The interfaces can be classified
+into VGA and non-VGA variants. The VGA ones are prefixed with virtio-vga
+or vhost-user-vga while the non-VGA ones are prefixed with virtio-gpu or
+vhost-user-gpu.
+
+The VGA ones always use the PCI interface, but for the non-VGA ones, the
+user can further pick between MMIO or PCI. For MMIO, the user can suffix
+the device name with -device, though vhost-user-gpu does not support MMIO.
+For PCI, the user can suffix it with -pci. Without these suffixes, the
+platform default will be chosen.
+
+virtio-gpu 2d
+-------------
+
+The default 2D backend only performs 2D operations. The guest needs to
+employ a software renderer for 3D graphics.
+
+Typically, the software renderer is provided by `Mesa`_ or `SwiftShader`_.
+Mesa's implementations (LLVMpipe, Lavapipe and virgl below) work out of box
+on typical modern Linux distributions.
+
+.. parsed-literal::
+    -device virtio-gpu
+
+.. _Mesa: https://www.mesa3d.org/
+.. _SwiftShader: https://github.com/google/swiftshader
+
+virtio-gpu virglrenderer
+------------------------
+
+When using virgl accelerated graphics mode in the guest, OpenGL API calls
+are translated into an intermediate representation (see `Gallium3D`_). The
+intermediate representation is communicated to the host and the
+`virglrenderer`_ library on the host translates the intermediate
+representation back to OpenGL API calls.
+
+.. parsed-literal::
+    -device virtio-gpu-gl
+
+.. _Gallium3D: https://www.freedesktop.org/wiki/Software/gallium/
+.. _virglrenderer: https://gitlab.freedesktop.org/virgl/virglrenderer/
+
+virtio-gpu rutabaga
+-------------------
+
+virtio-gpu can also leverage `rutabaga_gfx`_ to provide `gfxstream`_
+rendering and `Wayland display passthrough`_.  With the gfxstream rendering
+mode, GLES and Vulkan calls are forwarded to the host with minimal
+modification.
+
+The crosvm book provides directions on how to build a `gfxstream-enabled
+rutabaga`_ and launch a `guest Wayland proxy`_.
+
+This device does require host blob support (``hostmem`` field below). The
+``hostmem`` field specifies the size of virtio-gpu host memory window.
+This is typically between 256M and 8G.
+
+At least one capset (see colon separated ``capset_names`` below) must be
+specified when starting the device.  The currently supported
+``capset_names`` are ``gfxstream-vulkan`` and ``cross-domain`` on Linux
+guests. For Android guests, ``gfxstream-gles`` is also supported.
+
+The device will try to auto-detect the wayland socket path if the
+``cross-domain`` capset name is set.  The user may optionally specify
+``wayland_socket_path`` for non-standard paths.
+
+The ``wsi`` option can be set to ``surfaceless`` or ``headless``.
+Surfaceless doesn't create a native window surface, but does copy from the
+render target to the Pixman buffer if a virtio-gpu 2D hypercall is issued.
+Headless is like surfaceless, but doesn't copy to the Pixman buffer.
+Surfaceless is the default if ``wsi`` is not specified.
+
+.. parsed-literal::
+    -device virtio-gpu-rutabaga,capset_names=gfxstream-vulkan:cross-domain,
+       hostmem=8G,wayland_socket_path=/tmp/nonstandard/mock_wayland.sock,
+       wsi=headless
+
+.. _rutabaga_gfx: https://github.com/google/crosvm/blob/main/rutabaga_gfx/ffi/src/include/rutabaga_gfx_ffi.h
+.. _gfxstream: https://android.googlesource.com/platform/hardware/google/gfxstream/
+.. _Wayland display passthrough: https://www.youtube.com/watch?v=OZJiHMtIQ2M
+.. _gfxstream-enabled rutabaga: https://crosvm.dev/book/appendix/rutabaga_gfx.html
+.. _guest Wayland proxy: https://crosvm.dev/book/devices/wayland.html