[v2,00/17] PCI devices passthrough on Arm

Message

Rahul Singh Sept. 22, 2021, 11:34 a.m. UTC

Hello All,

The purpose of this patch series is to add PCI passthrough support to Xen on
Arm. PCI passthrough support on ARM is the collaboration work between EPAM and
ARM. ARM submitted the partial RFC [1][2] last year to get early feedback. We
tried to fix all the comments and added more features to this patch series.

Working POC with all the features can be found at [3]. Working POC is tested
on x86 so that there will be no regression on x86. Design presentation can be
found at [4]

PCI passthrough support is divided into different patches. This patch series
includes following features: 

Preparatory work to implement the PCI passthrough support for the ARM:
- Refactor MSI code.
- Fixed compilation error when HAS_PCI enabled for ARM.

Discovering PCI Host Bridge in XEN:
- PCI init to initialize the PCI driver.
- PCI host bridge discovery in XEN and map the PCI ECAM configuration space to
  the XEN memory.
- PCI access functions.

Discovering PCI devices:
- To support the PCI passthrough, XEN should be aware of the PCI
  devices.
- Hardware domain is in charge of doing the PCI enumeration and will discover
  the PCI devices and then communicate to the XEN via a hypercall to add the
  PCI devices in XEN.

Enable the existing x86 virtual PCI support for ARM:
- Add VPCI trap handler for each of the PCI device added for config space
  access.
- Register the trap handler in XEN for each of the host bridge PCI ECAM config
  space access.

Emulated PCI device tree node in libxl:
- Create a virtual PCI device tree node in libxl to enable the guest OS to
  discover the virtual PCI during guest boot.

This patch series does not inlcude the following features. Following features
will be send for review in the next version of the patch series once initial
patch series merged.

- VPCI support for DOMU guests (Non-identity mappings guest view of the BARs) 
- Virtual bus topology implementation
- IOMMU related changes (generic, SMMUv2, SMMUv3)
- MSI support for DOMU guests.
- Virual ITS support for DOMU guests

[1] https://lists.xenproject.org/archives/html/xen-devel/2020-07/msg01184.html
[2] https://lists.xenproject.org/archives/html/xen-devel/2020-07/threads.html#01184
[3] https://gitlab.com/rahsingh/xen-integration/-/commits/pci-passthrough-upstream-all
[4] https://static.sched.com/hosted_files/xen2021/e4/PCI_Device_Passthrough_On_Arm.pdf

Oleksandr Andrushchenko (1):
  xen/arm: Add support for Xilinx ZynqMP PCI host controller

Rahul Singh (16):
  xen/pci: Refactor MSI code that implements MSI functionality within
    XEN
  xen/pci: solve compilation error on ARM with HAS_PCI enabled
  xen/arm: solve compilation error on ARM with ACPI && HAS_PCI
  xen/arm: xc_domain_ioport_permission(..) not supported on ARM.
  xen/arm: Add PHYSDEVOP_pci_device_* support for ARM
  xen/device-tree: Add dt_property_read_variable_u32_array helper
  xen/device-tree: Add dt_property_read_u32_array helper
  xen/device-tree: Add dt_get_pci_domain_nr helper
  xen/arm: Add support for PCI init to initialize the PCI driver.
  xen/arm: Add cmdline boot option "pci-passthrough = <boolean>"
  xen/arm: PCI host bridge discovery within XEN on ARM
  xen:arm: Implement pci access functions
  xen/arm: Enable the existing x86 virtual PCI support for ARM.
  xen/arm: Transitional change to build HAS_VPCI on ARM.
  arm/libxl: Emulated PCI device tree node in libxl
  xen/arm: Add linux,pci-domain property for hwdom if not available.

 docs/misc/xen-command-line.pandoc   |   7 +
 tools/include/libxl.h               |   6 +
 tools/libs/ctrl/xc_domain.c         |   9 +
 tools/libs/light/libxl_arm.c        | 105 ++++++++++
 tools/libs/light/libxl_create.c     |   3 +
 tools/libs/light/libxl_types.idl    |   1 +
 tools/xl/xl_parse.c                 |   2 +
 xen/arch/arm/Makefile               |   2 +
 xen/arch/arm/device.c               |   2 +
 xen/arch/arm/domain.c               |   6 +-
 xen/arch/arm/domain_build.c         |  19 ++
 xen/arch/arm/pci/Makefile           |   7 +
 xen/arch/arm/pci/ecam.c             |  60 ++++++
 xen/arch/arm/pci/pci-access.c       | 141 +++++++++++++
 xen/arch/arm/pci/pci-host-common.c  | 294 ++++++++++++++++++++++++++++
 xen/arch/arm/pci/pci-host-generic.c |  42 ++++
 xen/arch/arm/pci/pci-host-zynqmp.c  |  58 ++++++
 xen/arch/arm/pci/pci.c              | 101 ++++++++++
 xen/arch/arm/physdev.c              |   5 +-
 xen/arch/arm/vpci.c                 | 102 ++++++++++
 xen/arch/arm/vpci.h                 |  36 ++++
 xen/arch/x86/Kconfig                |   1 +
 xen/arch/x86/physdev.c              |  50 +----
 xen/arch/x86/x86_64/physdev.c       |   4 +-
 xen/common/Makefile                 |   1 +
 xen/common/device_tree.c            |  73 +++++++
 xen/common/domain.c                 |   2 +-
 xen/common/physdev.c                |  87 ++++++++
 xen/drivers/passthrough/Makefile    |   1 +
 xen/drivers/passthrough/arm/iommu.c |   9 +
 xen/drivers/passthrough/msi.c       |  83 ++++++++
 xen/drivers/passthrough/pci.c       |  69 +++----
 xen/drivers/pci/Kconfig             |   4 +
 xen/drivers/vpci/Makefile           |   3 +-
 xen/drivers/vpci/header.c           |   2 +
 xen/include/asm-arm/device.h        |   3 +
 xen/include/asm-arm/domain.h        |   8 +-
 xen/include/asm-arm/hypercall.h     |   2 -
 xen/include/asm-arm/numa.h          |   5 +
 xen/include/asm-arm/pci.h           | 122 +++++++++++-
 xen/include/asm-x86/hypercall.h     |   9 +-
 xen/include/asm-x86/pci.h           |  10 +-
 xen/include/public/arch-arm.h       |  17 ++
 xen/include/public/domctl.h         |   4 +-
 xen/include/xen/device_tree.h       |  74 +++++++
 xen/include/xen/hypercall.h         |   8 +
 xen/include/xen/msi.h               |  43 ++++
 xen/include/xen/pci.h               |   2 +
 xen/xsm/flask/hooks.c               |   8 +-
 49 files changed, 1597 insertions(+), 115 deletions(-)
 create mode 100644 xen/arch/arm/pci/Makefile
 create mode 100644 xen/arch/arm/pci/ecam.c
 create mode 100644 xen/arch/arm/pci/pci-access.c
 create mode 100644 xen/arch/arm/pci/pci-host-common.c
 create mode 100644 xen/arch/arm/pci/pci-host-generic.c
 create mode 100644 xen/arch/arm/pci/pci-host-zynqmp.c
 create mode 100644 xen/arch/arm/pci/pci.c
 create mode 100644 xen/arch/arm/vpci.c
 create mode 100644 xen/arch/arm/vpci.h
 create mode 100644 xen/common/physdev.c
 create mode 100644 xen/drivers/passthrough/msi.c
 create mode 100644 xen/include/xen/msi.h