| Message ID | 20201109125031.26409-9-andr2000@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | ARM PCI passthrough configuration and vPCI | expand |
On Mon, Nov 09, 2020 at 02:50:29PM +0200, Oleksandr Andrushchenko wrote: > From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> > > Non-ECAM host bridges in hwdom go directly to PCI config space, > not through vpci (they use their specific method for accessing PCI > configuration, e.g. dedicated registers etc.). Thus hwdom's vpci BARs are > never updated via vPCI MMIO handlers, so implement a dedicated method > for a PCI host bridge, so it has a chance to update the initial state of > the device BARs. > > Note, we rely on the fact that control/hardware domain will not update > physical BAR locations for the given devices. This is quite ugly. I'm looking at the commit that implements the hook for R-Car and I'm having trouble seeing how that's different from the way we would normally read the BAR addresses. I think this should likely be paired with the actual implementation of a hook, or else it's hard to tell whether it really needed or not. Thanks, Roger.
On 11/12/20 11:56 AM, Roger Pau Monné wrote: > On Mon, Nov 09, 2020 at 02:50:29PM +0200, Oleksandr Andrushchenko wrote: >> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >> >> Non-ECAM host bridges in hwdom go directly to PCI config space, >> not through vpci (they use their specific method for accessing PCI >> configuration, e.g. dedicated registers etc.). Thus hwdom's vpci BARs are >> never updated via vPCI MMIO handlers, so implement a dedicated method >> for a PCI host bridge, so it has a chance to update the initial state of >> the device BARs. >> >> Note, we rely on the fact that control/hardware domain will not update >> physical BAR locations for the given devices. > This is quite ugly. It is > > I'm looking at the commit that implements the hook for R-Car and I'm > having trouble seeing how that's different from the way we would > normally read the BAR addresses. Ok, please see my comment on patch [06/10]. In short: when a PCI device is *added* we call init_bars and at that time BARs are not assigned on ARM yet. But, if we move init_bars to the point when a device *assigned* then it will work? And this code will go away > > I think this should likely be paired with the actual implementation of > a hook, or else it's hard to tell whether it really needed or not. Yes, if we move to device assign then it won't be needed: have to check that > > Thanks, Roger. Thank you, Oleksandr
On 09.11.2020 13:50, Oleksandr Andrushchenko wrote: > From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> > > Non-ECAM host bridges in hwdom go directly to PCI config space, > not through vpci (they use their specific method for accessing PCI > configuration, e.g. dedicated registers etc.). And access to these dedicated registers can't be intercepted? It would seem to me that if so, such a platform is not capable of being virtualized (without cooperation by all the domains in possession of devices). Jan
On 11/13/20 12:29 PM, Jan Beulich wrote: > On 09.11.2020 13:50, Oleksandr Andrushchenko wrote: >> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >> >> Non-ECAM host bridges in hwdom go directly to PCI config space, >> not through vpci (they use their specific method for accessing PCI >> configuration, e.g. dedicated registers etc.). > And access to these dedicated registers can't be intercepted? It can. But then you have to fully emulate that bridge, e.g. "if we write A to regB and after that write C to regZ then it means we are accessing config space. If we write...." I mean this would need lots of code in Xen to achieve that > It > would seem to me that if so, such a platform is not capable of > being virtualized (without cooperation by all the domains in > possession of devices). Guest domains always use an emulated ECAM bridge and are easily trapped and emulated > > Jan
On 13.11.2020 11:39, Oleksandr Andrushchenko wrote: > > On 11/13/20 12:29 PM, Jan Beulich wrote: >> On 09.11.2020 13:50, Oleksandr Andrushchenko wrote: >>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >>> >>> Non-ECAM host bridges in hwdom go directly to PCI config space, >>> not through vpci (they use their specific method for accessing PCI >>> configuration, e.g. dedicated registers etc.). >> And access to these dedicated registers can't be intercepted? > > It can. But then you have to fully emulate that bridge, e.g. > > "if we write A to regB and after that write C to regZ then it > > means we are accessing config space. If we write...." Sounds pretty much like the I/O port based access mechanism on x86, which also has some sort of "enable". Of course, I/O port accesses are particularly easy to intercept and handle... > I mean this would need lots of code in Xen to achieve that Possibly, but look at the amount of code we have in Xen on the x86 side to handle MCFG writes by Dom0. Jan
On 11/13/20 12:47 PM, Jan Beulich wrote: > On 13.11.2020 11:39, Oleksandr Andrushchenko wrote: >> On 11/13/20 12:29 PM, Jan Beulich wrote: >>> On 09.11.2020 13:50, Oleksandr Andrushchenko wrote: >>>> From: Oleksandr Andrushchenko <oleksandr_andrushchenko@epam.com> >>>> >>>> Non-ECAM host bridges in hwdom go directly to PCI config space, >>>> not through vpci (they use their specific method for accessing PCI >>>> configuration, e.g. dedicated registers etc.). >>> And access to these dedicated registers can't be intercepted? >> It can. But then you have to fully emulate that bridge, e.g. >> >> "if we write A to regB and after that write C to regZ then it >> >> means we are accessing config space. If we write...." > Sounds pretty much like the I/O port based access mechanism on > x86, which also has some sort of "enable". Of course, I/O port > accesses are particularly easy to intercept and handle... Yes, it has somewhat similar idea > >> I mean this would need lots of code in Xen to achieve that > Possibly, but look at the amount of code we have in Xen on the > x86 side to handle MCFG writes by Dom0. But MCFG is handled the same way for all x86 machines, right? And here I'll have to have a SoC specific code, e.g. a specific driver > > Jan Thank you, Oleksandr
diff --git a/xen/arch/arm/pci/pci-host-common.c b/xen/arch/arm/pci/pci-host-common.c index b6c4d7b636b1..5f4239afa41f 100644 --- a/xen/arch/arm/pci/pci-host-common.c +++ b/xen/arch/arm/pci/pci-host-common.c @@ -250,6 +250,19 @@ int pci_host_bridge_update_mappings(struct domain *d) return pci_host_iterate_bridges(d, pci_host_bridge_update_mapping); } +void pci_host_bridge_update_bar_header(const struct pci_dev *pdev, + struct vpci_header *header) +{ + struct pci_host_bridge *bridge; + + bridge = pci_find_host_bridge(pdev->seg, pdev->bus); + if ( unlikely(!bridge) ) + return; + + if ( bridge->ops->update_bar_header ) + bridge->ops->update_bar_header(pdev, header); +} + /* * Local variables: * mode: C diff --git a/xen/drivers/vpci/header.c b/xen/drivers/vpci/header.c index 7dc7c70e24f2..1f326c894d16 100644 --- a/xen/drivers/vpci/header.c +++ b/xen/drivers/vpci/header.c @@ -77,7 +77,14 @@ static struct vpci_header *get_vpci_header(struct domain *d, if ( !is_hardware_domain(d) ) { struct vpci_header *hwdom_header = get_hwdom_vpci_header(pdev); - +#ifdef CONFIG_ARM + /* + * Non-ECAM host bridges in hwdom go directly to PCI + * config space, not through vpci. Thus hwdom's vpci BARs are + * never updated. + */ + pci_host_bridge_update_bar_header(pdev, hwdom_header); +#endif /* Make a copy of the hwdom's BARs as the initial state for vBARs. */ memcpy(header, hwdom_header, sizeof(*header)); } diff --git a/xen/include/asm-arm/pci.h b/xen/include/asm-arm/pci.h index d94e8a6628de..723b2a99b6e1 100644 --- a/xen/include/asm-arm/pci.h +++ b/xen/include/asm-arm/pci.h @@ -60,6 +60,9 @@ struct pci_config_window { /* Forward declaration as pci_host_bridge and pci_ops depend on each other. */ struct pci_host_bridge; +struct pci_dev; +struct vpci_header; + struct pci_ops { int (*read)(struct pci_host_bridge *bridge, uint32_t sbdf, int where, int size, u32 *val); @@ -69,6 +72,8 @@ struct pci_ops { int (*register_mmio_handler)(struct domain *d, struct pci_host_bridge *bridge, const struct mmio_handler_ops *ops); + void (*update_bar_header)(const struct pci_dev *pdev, + struct vpci_header *header); }; /* @@ -110,6 +115,9 @@ int pci_host_iterate_bridges(struct domain *d, int (*clb)(struct domain *d, struct pci_host_bridge *bridge)); int pci_host_bridge_update_mappings(struct domain *d); +void pci_host_bridge_update_bar_header(const struct pci_dev *pdev, + struct vpci_header *header); + #else /*!CONFIG_ARM_PCI*/ struct arch_pci_dev { }; static inline void pci_init(void) { }