| Message ID | 20240924162359.1390487-3-edgar.iglesias@gmail.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | xen/arm: Add Virtio-PCI for dom0less on ARM | expand |
Hi Edgar, On 24/09/2024 17:23, Edgar E. Iglesias wrote: > From: "Edgar E. Iglesias" <edgar.iglesias@amd.com> > > Reserve memory ranges and interrupt lines for an externally > emulated PCI controller (e.g by QEMU) dedicated to hosting > Virtio devices and potentially other emulated devices. > > Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> > --- > xen/include/public/arch-arm.h | 17 +++++++++++++++++ > 1 file changed, 17 insertions(+) > > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h > index e19f0251a6..654b827715 100644 > --- a/xen/include/public/arch-arm.h > +++ b/xen/include/public/arch-arm.h > @@ -494,6 +494,20 @@ typedef uint64_t xen_callback_t; > #define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 952GB of RAM @ 8GB */ > #define GUEST_RAM1_SIZE xen_mk_ullong(0xee00000000) > > +/* Virtio PCI - Ordered by decreasing size to keep things aligned */ > +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_TYPE xen_mk_ullong(0x43000000) > +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_BASE xen_mk_ullong(0x0f000000000) > +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000) > + > +#define GUEST_VIRTIO_PCI_ECAM_BASE (GUEST_VIRTIO_PCI_PREFETCH_MEM_BASE + \ > + GUEST_VIRTIO_PCI_PREFETCH_MEM_SIZE) > +#define GUEST_VIRTIO_PCI_ECAM_SIZE xen_mk_ullong(0x10000000) > + > +#define GUEST_VIRTIO_PCI_MEM_TYPE xen_mk_ullong(0x02000000) > +#define GUEST_VIRTIO_PCI_MEM_BASE (GUEST_VIRTIO_PCI_ECAM_BASE + \ > + GUEST_VIRTIO_PCI_ECAM_SIZE) > +#define GUEST_VIRTIO_PCI_MEM_SIZE xen_mk_ullong(0x00002000000) Why is this specific to virtio PCI? However, I am not entirely convinced we should have a second PCI hostbridge exposed to the guest for a few reasons: 1. This require to reserve yet another range in the address space (could be solved with a more dynamic layout) 2. From your instructions, the guest needs to explicitly do a PCI rescan. So rather than having a second hostbridge, have you considered to extend the existing hostbridge (implemented in Xen) to support a mix of physical PCI device and virtual one? Cheers,
On Tue, Sep 24, 2024 at 05:35:20PM +0100, Julien Grall wrote: > Hi Edgar, > > On 24/09/2024 17:23, Edgar E. Iglesias wrote: > > From: "Edgar E. Iglesias" <edgar.iglesias@amd.com> > > > > Reserve memory ranges and interrupt lines for an externally > > emulated PCI controller (e.g by QEMU) dedicated to hosting > > Virtio devices and potentially other emulated devices. > > > > Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> > > --- > > xen/include/public/arch-arm.h | 17 +++++++++++++++++ > > 1 file changed, 17 insertions(+) > > > > diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h > > index e19f0251a6..654b827715 100644 > > --- a/xen/include/public/arch-arm.h > > +++ b/xen/include/public/arch-arm.h > > @@ -494,6 +494,20 @@ typedef uint64_t xen_callback_t; > > #define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 952GB of RAM @ 8GB */ > > #define GUEST_RAM1_SIZE xen_mk_ullong(0xee00000000) > > +/* Virtio PCI - Ordered by decreasing size to keep things aligned */ > > +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_TYPE xen_mk_ullong(0x43000000) > > +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_BASE xen_mk_ullong(0x0f000000000) > > +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000) > > + > > +#define GUEST_VIRTIO_PCI_ECAM_BASE (GUEST_VIRTIO_PCI_PREFETCH_MEM_BASE + \ > > + GUEST_VIRTIO_PCI_PREFETCH_MEM_SIZE) > > +#define GUEST_VIRTIO_PCI_ECAM_SIZE xen_mk_ullong(0x10000000) > > + > > +#define GUEST_VIRTIO_PCI_MEM_TYPE xen_mk_ullong(0x02000000) > > +#define GUEST_VIRTIO_PCI_MEM_BASE (GUEST_VIRTIO_PCI_ECAM_BASE + \ > > + GUEST_VIRTIO_PCI_ECAM_SIZE) > > +#define GUEST_VIRTIO_PCI_MEM_SIZE xen_mk_ullong(0x00002000000) > > Why is this specific to virtio PCI? However, I am not entirely convinced we > should have a second PCI hostbridge exposed to the guest for a few reasons: > 1. This require to reserve yet another range in the address space (could > be solved with a more dynamic layout) > 2. From your instructions, the guest needs to explicitly do a PCI rescan. > > So rather than having a second hostbridge, have you considered to extend the > existing hostbridge (implemented in Xen) to support a mix of physical PCI > device and virtual one? > Thanks Julien, It's briefly come up in a couple of discussions but I haven't looked carefully at it. It is a good idea and it's probably worth prototyping to see what the gaps are in hypercall interfaces, QEMU support etc. Cheers, Edgar
Hi Edgar, On 24/09/2024 18:11, Edgar E. Iglesias wrote: > On Tue, Sep 24, 2024 at 05:35:20PM +0100, Julien Grall wrote: >> Hi Edgar, >> >> On 24/09/2024 17:23, Edgar E. Iglesias wrote: >>> From: "Edgar E. Iglesias" <edgar.iglesias@amd.com> >>> >>> Reserve memory ranges and interrupt lines for an externally >>> emulated PCI controller (e.g by QEMU) dedicated to hosting >>> Virtio devices and potentially other emulated devices. >>> >>> Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> >>> --- >>> xen/include/public/arch-arm.h | 17 +++++++++++++++++ >>> 1 file changed, 17 insertions(+) >>> >>> diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h >>> index e19f0251a6..654b827715 100644 >>> --- a/xen/include/public/arch-arm.h >>> +++ b/xen/include/public/arch-arm.h >>> @@ -494,6 +494,20 @@ typedef uint64_t xen_callback_t; >>> #define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 952GB of RAM @ 8GB */ >>> #define GUEST_RAM1_SIZE xen_mk_ullong(0xee00000000) >>> +/* Virtio PCI - Ordered by decreasing size to keep things aligned */ >>> +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_TYPE xen_mk_ullong(0x43000000) >>> +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_BASE xen_mk_ullong(0x0f000000000) >>> +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000) >>> + >>> +#define GUEST_VIRTIO_PCI_ECAM_BASE (GUEST_VIRTIO_PCI_PREFETCH_MEM_BASE + \ >>> + GUEST_VIRTIO_PCI_PREFETCH_MEM_SIZE) >>> +#define GUEST_VIRTIO_PCI_ECAM_SIZE xen_mk_ullong(0x10000000) >>> + >>> +#define GUEST_VIRTIO_PCI_MEM_TYPE xen_mk_ullong(0x02000000) >>> +#define GUEST_VIRTIO_PCI_MEM_BASE (GUEST_VIRTIO_PCI_ECAM_BASE + \ >>> + GUEST_VIRTIO_PCI_ECAM_SIZE) >>> +#define GUEST_VIRTIO_PCI_MEM_SIZE xen_mk_ullong(0x00002000000) >> >> Why is this specific to virtio PCI? However, I am not entirely convinced we >> should have a second PCI hostbridge exposed to the guest for a few reasons: >> 1. This require to reserve yet another range in the address space (could >> be solved with a more dynamic layout) >> 2. From your instructions, the guest needs to explicitly do a PCI rescan. Another big advantage I forgot to mention is disaggregation. In a world where the hostbridge is handled in Xen, you could have a PCI device emulated by dom0 while the other is emulated by a stub domain. >> >> So rather than having a second hostbridge, have you considered to extend the >> existing hostbridge (implemented in Xen) to support a mix of physical PCI >> device and virtual one? >> > > Thanks Julien, > > It's briefly come up in a couple of discussions but I haven't looked > carefully at it. It is a good idea and it's probably worth prototyping > to see what the gaps are in hypercall interfaces, QEMU support etc. I also vaguely recall to discuss it on xen-devel. But I couldn't find the discussion... :(. I think all the hypercalls should be there but will require some plumbing in Xen on Arm. QEMU should be able to request Xen to forward configuration access for a given PCI device (see XEN_DMOP_IO_RANGE_PCI). They will then be forwarded to QEMU using IOREQ_TYPE_PCI_CONFIG. We also have an hypercall to be able to inject interrupts from QEMU (see XEN_DMOP_set_irq_level). Cheers,
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h index e19f0251a6..654b827715 100644 --- a/xen/include/public/arch-arm.h +++ b/xen/include/public/arch-arm.h @@ -494,6 +494,20 @@ typedef uint64_t xen_callback_t; #define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 952GB of RAM @ 8GB */ #define GUEST_RAM1_SIZE xen_mk_ullong(0xee00000000) +/* Virtio PCI - Ordered by decreasing size to keep things aligned */ +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_TYPE xen_mk_ullong(0x43000000) +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_BASE xen_mk_ullong(0x0f000000000) +#define GUEST_VIRTIO_PCI_PREFETCH_MEM_SIZE xen_mk_ullong(0x100000000) + +#define GUEST_VIRTIO_PCI_ECAM_BASE (GUEST_VIRTIO_PCI_PREFETCH_MEM_BASE + \ + GUEST_VIRTIO_PCI_PREFETCH_MEM_SIZE) +#define GUEST_VIRTIO_PCI_ECAM_SIZE xen_mk_ullong(0x10000000) + +#define GUEST_VIRTIO_PCI_MEM_TYPE xen_mk_ullong(0x02000000) +#define GUEST_VIRTIO_PCI_MEM_BASE (GUEST_VIRTIO_PCI_ECAM_BASE + \ + GUEST_VIRTIO_PCI_ECAM_SIZE) +#define GUEST_VIRTIO_PCI_MEM_SIZE xen_mk_ullong(0x00002000000) + #define GUEST_RAM_BASE GUEST_RAM0_BASE /* Lowest RAM address */ /* Largest amount of actual RAM, not including holes */ #define GUEST_RAM_MAX (GUEST_RAM0_SIZE + GUEST_RAM1_SIZE) @@ -529,6 +543,9 @@ typedef uint64_t xen_callback_t; #define GUEST_FFA_NOTIF_PEND_INTR_ID 8 #define GUEST_FFA_SCHEDULE_RECV_INTR_ID 9 +#define GUEST_VIRTIO_PCI_SPI_FIRST 44 +#define GUEST_VIRTIO_PCI_SPI_LAST 48 + /* PSCI functions */ #define PSCI_cpu_suspend 0 #define PSCI_cpu_off 1