[00/10,RFC] ARM PCI passthrough configuration and vPCI

Message

Oleksandr Andrushchenko Nov. 9, 2020, 12:50 p.m. UTC

From: Oleksandr Andrushchenko <[e-mail]>

Hello, all!

This is an RFC and an attempt to understand the best way to progress with ARM
PCI passthrough configuration. This includes, but not limited to, configuration
of assignable PCI devices, (legacy IRQs, MSI/MSI-X are not yet supported), MMIO
etc.

This is based on the original RFC from ARM [1] and bits of the possible
configuration approaches were discussed before [2]. So, I tried to implement
something, so we can discuss it in more detail. (Rahul, Bertrand: if you are
interested we can discuss all this in detail, so we can use this as a part of
ARM PCI passthrough solution).

This is all work in progress, so without having some other minor patches one
won’t be able to run that, but still the patches show the direction, which
should be fine for the RFC. Those interested in the full working example I have
created a branch [3], but please note that this was fully tested on R-Car Gen3
platform only which has a non-ECAM PCI host controller and only partially it
was tested on QEMU (without running guest domains, Domain-0 only).

In this RFC I only submit some patches which I would like to get community's
view on.  I will highlight some of them below and the rest is documented in
their commit messages:

1. [WORKAROUND] xen/arm: Update hwdom's p2m to trap ECAM space

This is a workaround to be able to trap ECAM address space in Domain-0 which is
normally not possible because PCI host bridge is mapped into Domain-0, so there
is a need in some special handling of such devices. I have discussed this with
Julien on IRC, but haven't implemented anything of production quality yet.

2. arm/pci: Maintain PCI assignable list

This patch needs a decision on pci-back use. As of now what is not covered is
the assignment of legacy IRQs (MMIOs and MSIs are handled by Xen without the
toolstack's help - am I right here?). MMIOs are assigned by vPCI code.  We
discussed [2] a possibility to run a "limited" version of the pci-back driver
for ARM, but I’d like to bring back that discussion here as it seems that only
some bits of the pci-back may now be used, so profit of having pci-back
in the picture is not clear.

3. vpci: Make every domain handle its own BARs

This is a big change in vPCI code which allows non-identity mappings for guest
domains. This also handles MMIO configuration for the guests without using the
toolstack which does the same via reading PCI bus sysfs entries in Domain-0.
(Thank you Roger for making lots of thing clear for me.) This implements PCI
headers per domain.

4. vpci/arm: Allow updating BAR's header for non-ECAM bridges

This allows non-ECAM bridges, which are not trapped by vPCI for Domain-0/hwdom,
to update vPCI's view of the real values of the BARs. The assumption here is
that Domain-0/hwdom won't relocate BARs which is usually the case.


5. Some code is for R-Car Gen3 which is not ECAM compatible. It is good for the
demostration of where generic ARM PCI framwork should be changed to support
such controllers.

Thank you,
Oleksandr Andrushchenko

P.S. I would like to thank Roger, Juilien and Jan for their attention
and time.

[1] https://www.mail-archive.com/[e-mail]/msg77422.html
[2] https://www.mail-archive.com/[e-mail]/msg84452.html
[3] https://github.com/andr2000/xen/tree/vpci_rfc

Oleksandr Andrushchenko (10):
  pci/pvh: Allow PCI toolstack code run with PVH domains on ARM
  arm/pci: Maintain PCI assignable list
  xen/arm: Setup MMIO range trap handlers for hardware domain
  [WORKAROUND] xen/arm: Update hwdom's p2m to trap ECAM space
  xen/arm: Process pending vPCI map/unmap operations
  vpci: Make every domain handle its own BARs
  xen/arm: Do not hardcode phycial PCI device addresses
  vpci/arm: Allow updating BAR's header for non-ECAM bridges
  vpci/rcar: Implement vPCI.update_bar_header callback
  [HACK] vpci/rcar: Make vPCI know DomD is hardware domain

 tools/libxc/include/xenctrl.h         |   9 +
 tools/libxc/xc_domain.c               |   1 +
 tools/libxc/xc_misc.c                 |  46 ++++
 tools/libxl/Makefile                  |   8 +
 tools/libxl/libxl_pci.c               | 109 +++++++++-
 xen/arch/arm/domain_build.c           |  10 +-
 xen/arch/arm/pci/pci-host-common.c    |  44 ++++
 xen/arch/arm/pci/pci-host-generic.c   |  43 +++-
 xen/arch/arm/pci/pci-host-rcar-gen3.c |  69 ++++++
 xen/arch/arm/sysctl.c                 |  66 +++++-
 xen/arch/arm/traps.c                  |   6 +
 xen/arch/arm/vpci.c                   |  16 +-
 xen/drivers/passthrough/pci.c         |  93 +++++++++
 xen/drivers/vpci/header.c             | 289 +++++++++++++++++++++++---
 xen/drivers/vpci/vpci.c               |   1 +
 xen/include/asm-arm/pci.h             |  17 ++
 xen/include/public/arch-arm.h         |   9 +-
 xen/include/public/sysctl.h           |  40 ++++
 xen/include/xen/pci.h                 |  12 ++
 xen/include/xen/vpci.h                |  24 ++-
 20 files changed, 857 insertions(+), 55 deletions(-)