| Message ID | 20220606203325.110625-1-mjrosato@linux.ibm.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | KVM: s390: enable zPCI for interpretive execution | expand |
On 6/6/22 4:33 PM, Matthew Rosato wrote: > Enable interpretive execution of zPCI instructions + adapter interruption > forwarding for s390x KVM vfio-pci. This is done by triggering a routine > when the VFIO group is associated with the KVM guest, transmitting to > firmware a special token (GISA designation) to enable that specific guest > for interpretive execution on that zPCI device. Load/store interpreation > enablement is then controlled by userspace (based upon whether or not a > SHM bit is placed in the virtual function handle). Adapter Event > Notification interpretation is controlled from userspace via a new KVM > ioctl. > > By allowing intepretation of zPCI instructions and firmware delivery of > interrupts to guests, we can reduce the frequency of guest SIE exits for > zPCI. > > From the perspective of guest configuration, you passthrough zPCI devices > in the same manner as before, with intepretation support being used by > default if available in kernel+qemu. > > Will follow up with a link the most recent QEMU series. > Latest QEMU series: https://lore.kernel.org/kvm/20220606203614.110928-1-mjrosato@linux.ibm.com/
On 6/6/22 4:33 PM, Matthew Rosato wrote: > Enable interpretive execution of zPCI instructions + adapter interruption > forwarding for s390x KVM vfio-pci. This is done by triggering a routine > when the VFIO group is associated with the KVM guest, transmitting to > firmware a special token (GISA designation) to enable that specific guest > for interpretive execution on that zPCI device. Load/store interpreation > enablement is then controlled by userspace (based upon whether or not a > SHM bit is placed in the virtual function handle). Adapter Event > Notification interpretation is controlled from userspace via a new KVM > ioctl. > > By allowing intepretation of zPCI instructions and firmware delivery of > interrupts to guests, we can reduce the frequency of guest SIE exits for > zPCI. > > From the perspective of guest configuration, you passthrough zPCI devices > in the same manner as before, with intepretation support being used by > default if available in kernel+qemu. > > Will follow up with a link the most recent QEMU series. > > Changelog v8->v9: > - Rebase on top of 5.19-rc1, adjust ioctl and capability defines > - s/kzdev = 0/kzdev = NULL/ (Alex) > - rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason) > - rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason) > - make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore > errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason) > - remove notifier accidentally left in struct zpci_dev + associated > include statment (Jason) > - Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation' > based on discussion in QEMU thread. > Ping -- I'm hoping this can make the next merge window, but there are still 2 patches left without any review tag (16 & 17).
Am 27.06.22 um 22:57 schrieb Matthew Rosato: > On 6/6/22 4:33 PM, Matthew Rosato wrote: >> Enable interpretive execution of zPCI instructions + adapter interruption >> forwarding for s390x KVM vfio-pci. This is done by triggering a routine >> when the VFIO group is associated with the KVM guest, transmitting to >> firmware a special token (GISA designation) to enable that specific guest >> for interpretive execution on that zPCI device. Load/store interpreation >> enablement is then controlled by userspace (based upon whether or not a >> SHM bit is placed in the virtual function handle). Adapter Event >> Notification interpretation is controlled from userspace via a new KVM >> ioctl. >> >> By allowing intepretation of zPCI instructions and firmware delivery of >> interrupts to guests, we can reduce the frequency of guest SIE exits for >> zPCI. >> >> From the perspective of guest configuration, you passthrough zPCI devices >> in the same manner as before, with intepretation support being used by >> default if available in kernel+qemu. >> >> Will follow up with a link the most recent QEMU series. >> >> Changelog v8->v9: >> - Rebase on top of 5.19-rc1, adjust ioctl and capability defines >> - s/kzdev = 0/kzdev = NULL/ (Alex) >> - rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason) >> - rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason) >> - make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore >> errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason) >> - remove notifier accidentally left in struct zpci_dev + associated >> include statment (Jason) >> - Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation' >> based on discussion in QEMU thread. >> > > Ping -- I'm hoping this can make the next merge window, but there are still 2 patches left without any review tag (16 & 17). Yes, I will queue this (as is). Ideally you would rebase this on top of kvm/next but I can also do while applying. Let me know if you want to respin with the Nits from Pierre.
On 6/28/22 8:35 AM, Christian Borntraeger wrote: > Am 27.06.22 um 22:57 schrieb Matthew Rosato: >> On 6/6/22 4:33 PM, Matthew Rosato wrote: >>> Enable interpretive execution of zPCI instructions + adapter >>> interruption >>> forwarding for s390x KVM vfio-pci. This is done by triggering a routine >>> when the VFIO group is associated with the KVM guest, transmitting to >>> firmware a special token (GISA designation) to enable that specific >>> guest >>> for interpretive execution on that zPCI device. Load/store >>> interpreation >>> enablement is then controlled by userspace (based upon whether or not a >>> SHM bit is placed in the virtual function handle). Adapter Event >>> Notification interpretation is controlled from userspace via a new KVM >>> ioctl. >>> >>> By allowing intepretation of zPCI instructions and firmware delivery of >>> interrupts to guests, we can reduce the frequency of guest SIE exits for >>> zPCI. >>> >>> From the perspective of guest configuration, you passthrough zPCI >>> devices >>> in the same manner as before, with intepretation support being used by >>> default if available in kernel+qemu. >>> >>> Will follow up with a link the most recent QEMU series. >>> >>> Changelog v8->v9: >>> - Rebase on top of 5.19-rc1, adjust ioctl and capability defines >>> - s/kzdev = 0/kzdev = NULL/ (Alex) >>> - rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason) >>> - rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason) >>> - make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore >>> errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason) >>> - remove notifier accidentally left in struct zpci_dev + associated >>> include statment (Jason) >>> - Remove patch 'KVM: s390: introduce CPU feature for zPCI >>> Interpretation' >>> based on discussion in QEMU thread. >>> >> >> Ping -- I'm hoping this can make the next merge window, but there are >> still 2 patches left without any review tag (16 & 17). > > Yes, I will queue this (as is). Ideally you would rebase this on top of > kvm/next but I can also do while applying. > Let me know if you want to respin with the Nits from Pierre. Ah, sorry -- I assume you mean Paolo's kvm/next? I tried now and see some conflicts with the ioctl patch. Why don't I rebase on top of kvm/next along with these couple of changes from Pierre and send this as a v10 for you to queue. While at it, there's one other issue to be aware of -- There will also be small merge conflicts with a patch that just hit vfio-next, "vfio: de-extern-ify function prototypes" - My series already avoids adding externs to new prototypes, but adjacent code changes will cause a conflict with patches 10 and 17. Not sure what the best way to proceed there is. https://lore.kernel.org/kvm/165471414407.203056.474032786990662279.stgit@omen/
On Tue, Jun 28, 2022 at 09:40:01AM -0400, Matthew Rosato wrote: > On 6/28/22 8:35 AM, Christian Borntraeger wrote: > > Am 27.06.22 um 22:57 schrieb Matthew Rosato: > > > On 6/6/22 4:33 PM, Matthew Rosato wrote: > > > > Enable interpretive execution of zPCI instructions + adapter > > > > interruption > > > > forwarding for s390x KVM vfio-pci. This is done by triggering a routine > > > > when the VFIO group is associated with the KVM guest, transmitting to > > > > firmware a special token (GISA designation) to enable that > > > > specific guest > > > > for interpretive execution on that zPCI device. Load/store > > > > interpreation > > > > enablement is then controlled by userspace (based upon whether or not a > > > > SHM bit is placed in the virtual function handle). Adapter Event > > > > Notification interpretation is controlled from userspace via a new KVM > > > > ioctl. > > > > > > > > By allowing intepretation of zPCI instructions and firmware delivery of > > > > interrupts to guests, we can reduce the frequency of guest SIE exits for > > > > zPCI. > > > > > > > > From the perspective of guest configuration, you passthrough > > > > zPCI devices > > > > in the same manner as before, with intepretation support being used by > > > > default if available in kernel+qemu. > > > > > > > > Will follow up with a link the most recent QEMU series. > > > > > > > > Changelog v8->v9: > > > > - Rebase on top of 5.19-rc1, adjust ioctl and capability defines > > > > - s/kzdev = 0/kzdev = NULL/ (Alex) > > > > - rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason) > > > > - rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason) > > > > - make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore > > > > errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason) > > > > - remove notifier accidentally left in struct zpci_dev + associated > > > > include statment (Jason) > > > > - Remove patch 'KVM: s390: introduce CPU feature for zPCI > > > > Interpretation' > > > > based on discussion in QEMU thread. > > > > > > > > > > Ping -- I'm hoping this can make the next merge window, but there > > > are still 2 patches left without any review tag (16 & 17). > > > > Yes, I will queue this (as is). Ideally you would rebase this on top of > > kvm/next but I can also do while applying. > > Let me know if you want to respin with the Nits from Pierre. > > Ah, sorry -- I assume you mean Paolo's kvm/next? I tried now and see some > conflicts with the ioctl patch. > > Why don't I rebase on top of kvm/next along with these couple of changes > from Pierre and send this as a v10 for you to queue. > > While at it, there's one other issue to be aware of -- There will also be > small merge conflicts with a patch that just hit vfio-next, "vfio: > de-extern-ify function prototypes" - My series already avoids adding externs > to new prototypes, but adjacent code changes will cause a conflict with > patches 10 and 17. > > Not sure what the best way to proceed there is. You should use a branch based on vfio-next and send a Git PR to Christian and Alex Jason
Am 28.06.22 um 15:40 schrieb Matthew Rosato: > On 6/28/22 8:35 AM, Christian Borntraeger wrote: >> Am 27.06.22 um 22:57 schrieb Matthew Rosato: >>> On 6/6/22 4:33 PM, Matthew Rosato wrote: >>>> Enable interpretive execution of zPCI instructions + adapter interruption >>>> forwarding for s390x KVM vfio-pci. This is done by triggering a routine >>>> when the VFIO group is associated with the KVM guest, transmitting to >>>> firmware a special token (GISA designation) to enable that specific guest >>>> for interpretive execution on that zPCI device. Load/store interpreation >>>> enablement is then controlled by userspace (based upon whether or not a >>>> SHM bit is placed in the virtual function handle). Adapter Event >>>> Notification interpretation is controlled from userspace via a new KVM >>>> ioctl. >>>> >>>> By allowing intepretation of zPCI instructions and firmware delivery of >>>> interrupts to guests, we can reduce the frequency of guest SIE exits for >>>> zPCI. >>>> >>>> From the perspective of guest configuration, you passthrough zPCI devices >>>> in the same manner as before, with intepretation support being used by >>>> default if available in kernel+qemu. >>>> >>>> Will follow up with a link the most recent QEMU series. >>>> >>>> Changelog v8->v9: >>>> - Rebase on top of 5.19-rc1, adjust ioctl and capability defines >>>> - s/kzdev = 0/kzdev = NULL/ (Alex) >>>> - rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason) >>>> - rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason) >>>> - make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore >>>> errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason) >>>> - remove notifier accidentally left in struct zpci_dev + associated >>>> include statment (Jason) >>>> - Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation' >>>> based on discussion in QEMU thread. >>>> >>> >>> Ping -- I'm hoping this can make the next merge window, but there are still 2 patches left without any review tag (16 & 17). >> >> Yes, I will queue this (as is). Ideally you would rebase this on top of kvm/next but I can also do while applying. >> Let me know if you want to respin with the Nits from Pierre. > > Ah, sorry -- I assume you mean Paolo's kvm/next? I tried now and see some conflicts with the ioctl patch. > > Why don't I rebase on top of kvm/next along with these couple of changes from Pierre and send this as a v10 for you to queue. > > While at it, there's one other issue to be aware of -- There will also be small merge conflicts with a patch that just hit vfio-next, "vfio: de-extern-ify function prototypes" - My series already avoids adding externs to new prototypes, but adjacent code changes will cause a conflict with patches 10 and 17. > > Not sure what the best way to proceed there is. > > https://lore.kernel.org/kvm/165471414407.203056.474032786990662279.stgit@omen/ I think Linus can sort out if the conflicts are trivial. As an alternative Alex could carry these patches, but then we have a merge conflict between him and KVM. Alex/Paolo, shall I do a topic branch that you both can merge?
Am 06.06.22 um 22:33 schrieb Matthew Rosato: > Enable interpretive execution of zPCI instructions + adapter interruption > forwarding for s390x KVM vfio-pci. This is done by triggering a routine > when the VFIO group is associated with the KVM guest, transmitting to > firmware a special token (GISA designation) to enable that specific guest > for interpretive execution on that zPCI device. Load/store interpreation > enablement is then controlled by userspace (based upon whether or not a > SHM bit is placed in the virtual function handle). Adapter Event > Notification interpretation is controlled from userspace via a new KVM > ioctl. > > By allowing intepretation of zPCI instructions and firmware delivery of > interrupts to guests, we can reduce the frequency of guest SIE exits for > zPCI. > > From the perspective of guest configuration, you passthrough zPCI devices > in the same manner as before, with intepretation support being used by > default if available in kernel+qemu. > > Will follow up with a link the most recent QEMU series. > > Changelog v8->v9: > - Rebase on top of 5.19-rc1, adjust ioctl and capability defines > - s/kzdev = 0/kzdev = NULL/ (Alex) > - rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason) > - rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason) > - make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore > errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason) > - remove notifier accidentally left in struct zpci_dev + associated > include statment (Jason) > - Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation' > based on discussion in QEMU thread. > > Matthew Rosato (21): > s390/sclp: detect the zPCI load/store interpretation facility > s390/sclp: detect the AISII facility > s390/sclp: detect the AENI facility > s390/sclp: detect the AISI facility > s390/airq: pass more TPI info to airq handlers > s390/airq: allow for airq structure that uses an input vector > s390/pci: externalize the SIC operation controls and routine > s390/pci: stash associated GISA designation > s390/pci: stash dtsm and maxstbl > vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM > KVM: s390: pci: add basic kvm_zdev structure > KVM: s390: pci: do initial setup for AEN interpretation > KVM: s390: pci: enable host forwarding of Adapter Event Notifications > KVM: s390: mechanism to enable guest zPCI Interpretation > KVM: s390: pci: provide routines for enabling/disabling interrupt > forwarding > KVM: s390: pci: add routines to start/stop interpretive execution > vfio-pci/zdev: add open/close device hooks > vfio-pci/zdev: add function handle to clp base capability > vfio-pci/zdev: different maxstbl for interpreted devices > KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices > MAINTAINERS: additional files related kvm s390 pci passthrough > > Documentation/virt/kvm/api.rst | 47 +++ > MAINTAINERS | 1 + > arch/s390/include/asm/airq.h | 7 +- > arch/s390/include/asm/kvm_host.h | 23 ++ > arch/s390/include/asm/pci.h | 11 + > arch/s390/include/asm/pci_clp.h | 9 +- > arch/s390/include/asm/pci_insn.h | 29 +- > arch/s390/include/asm/sclp.h | 4 + > arch/s390/include/asm/tpi.h | 13 + > arch/s390/kvm/Makefile | 1 + > arch/s390/kvm/interrupt.c | 96 ++++- > arch/s390/kvm/kvm-s390.c | 83 +++- > arch/s390/kvm/kvm-s390.h | 10 + > arch/s390/kvm/pci.c | 690 +++++++++++++++++++++++++++++++ > arch/s390/kvm/pci.h | 88 ++++ > arch/s390/pci/pci.c | 16 + > arch/s390/pci/pci_clp.c | 7 + > arch/s390/pci/pci_insn.c | 4 +- > arch/s390/pci/pci_irq.c | 48 ++- > drivers/s390/char/sclp_early.c | 4 + > drivers/s390/cio/airq.c | 12 +- > drivers/s390/cio/qdio_thinint.c | 6 +- > drivers/s390/crypto/ap_bus.c | 9 +- > drivers/s390/virtio/virtio_ccw.c | 6 +- > drivers/vfio/pci/Kconfig | 11 + > drivers/vfio/pci/Makefile | 2 +- > drivers/vfio/pci/vfio_pci_core.c | 10 +- > drivers/vfio/pci/vfio_pci_zdev.c | 35 +- > include/linux/sched/user.h | 3 +- > include/linux/vfio_pci_core.h | 12 +- > include/uapi/linux/kvm.h | 31 ++ > include/uapi/linux/vfio_zdev.h | 7 + > 32 files changed, 1279 insertions(+), 56 deletions(-) > create mode 100644 arch/s390/kvm/pci.c > create mode 100644 arch/s390/kvm/pci.h So I pulled this into a topic branch and will merge that into kvms390/next. We can merge this topic branch into vfio-next and/or s390-next when the conflicts get to complicated. While pulling I fixed up the numbers for the capability to #define KVM_CAP_S390_ZPCI_OP 221 and the doc number to 4.137 KVM_S390_ZPCI_OP to minize struggle when doing backports.
Enable interpretive execution of zPCI instructions + adapter interruption forwarding for s390x KVM vfio-pci. This is done by triggering a routine when the VFIO group is associated with the KVM guest, transmitting to firmware a special token (GISA designation) to enable that specific guest for interpretive execution on that zPCI device. Load/store interpreation enablement is then controlled by userspace (based upon whether or not a SHM bit is placed in the virtual function handle). Adapter Event Notification interpretation is controlled from userspace via a new KVM ioctl. By allowing intepretation of zPCI instructions and firmware delivery of interrupts to guests, we can reduce the frequency of guest SIE exits for zPCI. From the perspective of guest configuration, you passthrough zPCI devices in the same manner as before, with intepretation support being used by default if available in kernel+qemu. Will follow up with a link the most recent QEMU series. Changelog v8->v9: - Rebase on top of 5.19-rc1, adjust ioctl and capability defines - s/kzdev = 0/kzdev = NULL/ (Alex) - rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason) - rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason) - make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason) - remove notifier accidentally left in struct zpci_dev + associated include statment (Jason) - Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation' based on discussion in QEMU thread. Matthew Rosato (21): s390/sclp: detect the zPCI load/store interpretation facility s390/sclp: detect the AISII facility s390/sclp: detect the AENI facility s390/sclp: detect the AISI facility s390/airq: pass more TPI info to airq handlers s390/airq: allow for airq structure that uses an input vector s390/pci: externalize the SIC operation controls and routine s390/pci: stash associated GISA designation s390/pci: stash dtsm and maxstbl vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM KVM: s390: pci: add basic kvm_zdev structure KVM: s390: pci: do initial setup for AEN interpretation KVM: s390: pci: enable host forwarding of Adapter Event Notifications KVM: s390: mechanism to enable guest zPCI Interpretation KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding KVM: s390: pci: add routines to start/stop interpretive execution vfio-pci/zdev: add open/close device hooks vfio-pci/zdev: add function handle to clp base capability vfio-pci/zdev: different maxstbl for interpreted devices KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices MAINTAINERS: additional files related kvm s390 pci passthrough Documentation/virt/kvm/api.rst | 47 +++ MAINTAINERS | 1 + arch/s390/include/asm/airq.h | 7 +- arch/s390/include/asm/kvm_host.h | 23 ++ arch/s390/include/asm/pci.h | 11 + arch/s390/include/asm/pci_clp.h | 9 +- arch/s390/include/asm/pci_insn.h | 29 +- arch/s390/include/asm/sclp.h | 4 + arch/s390/include/asm/tpi.h | 13 + arch/s390/kvm/Makefile | 1 + arch/s390/kvm/interrupt.c | 96 ++++- arch/s390/kvm/kvm-s390.c | 83 +++- arch/s390/kvm/kvm-s390.h | 10 + arch/s390/kvm/pci.c | 690 +++++++++++++++++++++++++++++++ arch/s390/kvm/pci.h | 88 ++++ arch/s390/pci/pci.c | 16 + arch/s390/pci/pci_clp.c | 7 + arch/s390/pci/pci_insn.c | 4 +- arch/s390/pci/pci_irq.c | 48 ++- drivers/s390/char/sclp_early.c | 4 + drivers/s390/cio/airq.c | 12 +- drivers/s390/cio/qdio_thinint.c | 6 +- drivers/s390/crypto/ap_bus.c | 9 +- drivers/s390/virtio/virtio_ccw.c | 6 +- drivers/vfio/pci/Kconfig | 11 + drivers/vfio/pci/Makefile | 2 +- drivers/vfio/pci/vfio_pci_core.c | 10 +- drivers/vfio/pci/vfio_pci_zdev.c | 35 +- include/linux/sched/user.h | 3 +- include/linux/vfio_pci_core.h | 12 +- include/uapi/linux/kvm.h | 31 ++ include/uapi/linux/vfio_zdev.h | 7 + 32 files changed, 1279 insertions(+), 56 deletions(-) create mode 100644 arch/s390/kvm/pci.c create mode 100644 arch/s390/kvm/pci.h