| Message ID | 20240717110326.45230-1-phil@philjordan.eu (mailing list archive) |
|---|---|
| Headers | show |
| Series | hw/display/apple-gfx: New macOS PV Graphics device | expand |
On 2024/07/17 20:03, Phil Dennis-Jordan wrote: > This sequence of patches integrates the paravirtualised graphics device > implemented by macOS's ParavirtualizedGraphics.Framework into Qemu. > Combined with the guest drivers which ship with macOS versions 11 and up, > this allows the guest OS to use the host's GPU for hardware accelerated > 3D graphics, GPGPU compute (both using the 'Metal' graphics API), and > window compositing. > > Some background: > ---------------- > > The device exposed by the ParavirtualizedGraphics.Framework's (henceforth > PVG) public API consists of a PCI device with a single memory-mapped BAR; > the VMM is expected to pass reads and writes through to the framework, and > to forward interrupts emenating from it to the guest VM. > > The bulk of data exchange between host and guest occurs via shared memory, > however. For this purpose, PVG makes callbacks to VMM code for allocating, > mapping, unmapping, and deallocating "task" memory ranges. Each task > represents a contiguous host virtual address range, and PVG expects the > VMM to map specific guest system memory ranges to these host addresses via > subsequent map callbacks. Multiple tasks can exist at a time, each with > many mappings. > > Data is exchanged via an undocumented, Apple-proprietary protocol. The > PVG API only acts as a facilitator for establishing the communication > mechanism. This is perhaps not ideal, and among other things means it > only works on macOS hosts, but it's the only serious option we've got for > good performance and quality graphics with macOS guests at this time. > > The first iterations of this PVG integration into Qemu were developed > by Alexander Graf as part of his "vmapple" machine patch series for > supporting aarch64 macOS guests, and posted to qemu-devel in June and > August 2023: > > https://lore.kernel.org/all/20230830161425.91946-1-graf@amazon.com/T/ > > This integration mimics the "vmapple"/"apple-gfx" variant of the PVG device > used by Apple's own VMM, Virtualization.framework. This variant does not use > PCI but acts as a direct MMIO system device; there are two MMIO ranges, one > behaving identically to the PCI BAR, while the other's functionality is > exposed by private APIs in the PVG framework. It is only available on aarch64 > macOS hosts. > > I had prior to this simultaneously and independently developed my own PVG > integration for Qemu using the public PCI device APIs, with x86-64 and > corresponding macOS guests and hosts as the target. After some months of > use in production, I was slowly reviewing the code and readying it for > upstreaming around the time Alexander posted his vmapple patches. > > I ended up reviewing the vmapple PVG code in detail; I identified a number > of issues with it (mainly thanks to my prior trial-and-error working with > the framework) but overall I thought it a better basis for refinement > than my own version: > > - It implemented the vmapple variant of the device. I thought it better to > port the part I understood well (PCI variant) to this than trying to port > the part I didn't understand well (MMIO vmapple variant) to my own code. > - The code was already tidier than my own. > > It also became clear in out-of-band communication that Alexander would > probably not end up having the time to see the patch through to inclusion, > and was happy for me to start making changes and to integrate my PCI code. I think you also need to take over the base vmapple change because PVG cannot be tested without it; Only macOS can use PVG, and macOS requires vmapple in my understanding. Regards, Akihiko Odaki
On Sat 20. Jul 2024 at 16:42, Akihiko Odaki <akihiko.odaki@daynix.com> wrote: > > > It also became clear in out-of-band communication that Alexander would > > probably not end up having the time to see the patch through to > inclusion, > > and was happy for me to start making changes and to integrate my PCI > code. > > I think you also need to take over the base vmapple change because PVG > cannot be tested without it; Only macOS can use PVG, and macOS requires > vmapple in my understanding. The PCI device variant works fine with x86-64 macOS as the guest system. (Or technically any UEFI based guest as the bootrom comes with a UEFI frame buffer driver; it just won’t have any acceleration features unless you boot macOS.) If preferable I can leave out the vmapple/MMIO device variant (the -vmapple.m file) until the rest of the vmapple machine type modifications are ready. There still appears to be some kind of interrupt delivery issue with some devices on vmapple, so USB HID events are very slow. Or I can submit it as is if that’s not a dealbreaker. (How do I handle Alex’ unmodified patches though, as git send-email tries to send them from the patch author’s email address, which means the email rapidly gets shot down by DKIM mismatch or whatever?) Thanks, Phil
On 2024/07/21 0:16, Phil Dennis-Jordan wrote: > > > On Sat 20. Jul 2024 at 16:42, Akihiko Odaki <akihiko.odaki@daynix.com > <mailto:akihiko.odaki@daynix.com>> wrote: > > > > It also became clear in out-of-band communication that Alexander > would > > probably not end up having the time to see the patch through to > inclusion, > > and was happy for me to start making changes and to integrate my > PCI code. > > I think you also need to take over the base vmapple change because PVG > cannot be tested without it; Only macOS can use PVG, and macOS requires > vmapple in my understanding. > > > The PCI device variant works fine with x86-64 macOS as the guest system. > (Or technically any UEFI based guest as the bootrom comes with a UEFI > frame buffer driver; it just won’t have any acceleration features unless > you boot macOS.) I tried apple-gfx-pci with AArch64 and x86-64 EDK firmware on M2 MacBook Air, but it didn't work. I guess the bootrom does not work with Apple Silicon. > > If preferable I can leave out the vmapple/MMIO device variant (the > -vmapple.m file) until the rest of the vmapple machine type > modifications are ready. There still appears to be some kind of > interrupt delivery issue with some devices on vmapple, so USB HID events > are very slow. Or I can submit it as is if that’s not a dealbreaker. The vmapple variant should be omitted for now. It is fine for me to submit the vmapple variant with the other vmapple changes as long as that behavior is documented. > (How do I handle Alex’ unmodified patches though, as git send-email > tries to send them from the patch author’s email address, which means > the email rapidly gets shot down by DKIM mismatch or whatever?) I think you can set sendemail.from configuration or specify --from option. Regards, Akihiko Odaki
This sequence of patches integrates the paravirtualised graphics device implemented by macOS's ParavirtualizedGraphics.Framework into Qemu. Combined with the guest drivers which ship with macOS versions 11 and up, this allows the guest OS to use the host's GPU for hardware accelerated 3D graphics, GPGPU compute (both using the 'Metal' graphics API), and window compositing. Some background: ---------------- The device exposed by the ParavirtualizedGraphics.Framework's (henceforth PVG) public API consists of a PCI device with a single memory-mapped BAR; the VMM is expected to pass reads and writes through to the framework, and to forward interrupts emenating from it to the guest VM. The bulk of data exchange between host and guest occurs via shared memory, however. For this purpose, PVG makes callbacks to VMM code for allocating, mapping, unmapping, and deallocating "task" memory ranges. Each task represents a contiguous host virtual address range, and PVG expects the VMM to map specific guest system memory ranges to these host addresses via subsequent map callbacks. Multiple tasks can exist at a time, each with many mappings. Data is exchanged via an undocumented, Apple-proprietary protocol. The PVG API only acts as a facilitator for establishing the communication mechanism. This is perhaps not ideal, and among other things means it only works on macOS hosts, but it's the only serious option we've got for good performance and quality graphics with macOS guests at this time. The first iterations of this PVG integration into Qemu were developed by Alexander Graf as part of his "vmapple" machine patch series for supporting aarch64 macOS guests, and posted to qemu-devel in June and August 2023: https://lore.kernel.org/all/20230830161425.91946-1-graf@amazon.com/T/ This integration mimics the "vmapple"/"apple-gfx" variant of the PVG device used by Apple's own VMM, Virtualization.framework. This variant does not use PCI but acts as a direct MMIO system device; there are two MMIO ranges, one behaving identically to the PCI BAR, while the other's functionality is exposed by private APIs in the PVG framework. It is only available on aarch64 macOS hosts. I had prior to this simultaneously and independently developed my own PVG integration for Qemu using the public PCI device APIs, with x86-64 and corresponding macOS guests and hosts as the target. After some months of use in production, I was slowly reviewing the code and readying it for upstreaming around the time Alexander posted his vmapple patches. I ended up reviewing the vmapple PVG code in detail; I identified a number of issues with it (mainly thanks to my prior trial-and-error working with the framework) but overall I thought it a better basis for refinement than my own version: - It implemented the vmapple variant of the device. I thought it better to port the part I understood well (PCI variant) to this than trying to port the part I didn't understand well (MMIO vmapple variant) to my own code. - The code was already tidier than my own. It also became clear in out-of-band communication that Alexander would probably not end up having the time to see the patch through to inclusion, and was happy for me to start making changes and to integrate my PCI code. It's taken a while, but I'm happy with the result and think it will be a welcome addition for anyone running macOS VMs. What doesn't work: ------------------ * State (de-)serialisation and thus migration. There is no fundamental technical obstacle to this. PVG supports saving and loading device state. I have simply not had the resources to implement (and crucially, test it) it yet. * Setting the list of display modes via a property is currently only implemented on the PCI version, which is the only one readily testable without the out-of-tree vmapple patches. (See review notes for patch 7) * End-to-end GPU-only rendering. After the host GPU has rendered the guest's screen, the framebuffer is copied into a system memory buffer (surface). When using the Qemu Cocoa UI, this buffer is drawn by the CPU into a GPU texture used for hardware window compositing. It would be vastly more efficient to retain the Metal texture and pass it directly through to the Cocoa window. We currently have no mechanism for doing so; it would need to be similar to the end-to-end OpenGL rendering support, with the added complication that Metal textures are Objective-C types and would need to traverse the plain C code of the Qemu display subsystem. * Dirty region detection. Similarly, the whole framebuffer is marked modified even if there has only been a small change. This hurts network data volume when using VNC. * Multi-head support. PVG allows "connecting" more than one virtual display. This integration currently always uses exactly 1 display. * The vmapple/aarch64 variant of the device is only testable with Alexander's vmapple machine type patch set. I've been maintaining this out-of-tree and have made a few improvements, but it doesn't yet run smoothly. (Graphics work fine with this code, issues are with other devices.) I can push my current draft to a git forge if anyone wants to test with them. I'm definitely hoping to eventually resolve the remaining problems and submit a revised version of that patch set as well. I think we can live without these for the moment, and I'd prefer to work on them only if and when the baseline functionality has been merged. Patch review notes: ------------------- As requested, I have now squashed any fixes and refactorings into the orginal commit and changed that to be marked as co-authored. Brief meta-discussion of specific patches and groups of patches: 01: Alexander Graf's original patch, rebased on latest upstream and combined with a long list of improvements. 02: The PCI variant. This implements the PCI variant of the PVG device in addition to the original vmapple/mmio one. The PCI variant is intended for x86-64 macOS guests. 03-06: Various improvements on top of the baseline functionality. I've left these non-squashed as I think some of them might generate more feedback and iteration, and thus may distract from the main task of getting the core merged. 07: QOM property for specifying the display mode list (resolutions) the device will report to the guest. I checked other display devices and found none supported this, though I personally find it very useful. I'm wondering whether this should be a more generic feature optionally usable by any display device in Qemu? 08: Adding myself as maintainer for the PVG code, and reviewer for HVF. Changes since v1: ----------------- * Squashed most of the commits into the first one, including the dpy_cursor_define_supported removal as that API has now disappeared from upstream. * Added comments explaining the reasons for different MMIO write handling on aarch64 vs x86-64. * Integrated cursor image RGBA<->BGRA conversion which I'd missed in the v1 patch set. (This fixes the colours on non-greyscale guest cursors.) * Removed a stray macOS 12 availability check. (These are no longer needed as Qemu 9.1 removes support for macOS 11 and older hosts.) Phil Dennis-Jordan (8): hw/display/apple-gfx: Introduce ParavirtualizedGraphics.Framework support hw/display/apple-gfx: Adds PCI implementation ui/cocoa: Adds non-app runloop on main thread mode hw/display/apple-gfx: Implements texture syncing for non-UMA GPUs hw/display/apple-gfx: Replaces magic number with queried MMIO length hw/display/apple-gfx: Host GPU picking improvements hw/display/apple-gfx: Adds configurable mode list MAINTAINERS: Add myself as maintainer for apple-gfx, reviewer for HVF MAINTAINERS | 7 + hw/display/Kconfig | 13 + hw/display/apple-gfx-pci.m | 162 ++++++++ hw/display/apple-gfx-vmapple.m | 194 ++++++++++ hw/display/apple-gfx.h | 72 ++++ hw/display/apple-gfx.m | 681 +++++++++++++++++++++++++++++++++ hw/display/meson.build | 3 + hw/display/trace-events | 26 ++ include/qemu-main.h | 2 + meson.build | 4 + ui/cocoa.m | 15 +- 11 files changed, 1177 insertions(+), 2 deletions(-) create mode 100644 hw/display/apple-gfx-pci.m create mode 100644 hw/display/apple-gfx-vmapple.m create mode 100644 hw/display/apple-gfx.h create mode 100644 hw/display/apple-gfx.m