diff mbox series

[v2,12/15] PCI/MSI: Let PCI host bridges declare their reliance on MSI domains

Message ID 20210322184614.802565-13-maz@kernel.org (mailing list archive)
State New
Headers show
Series PCI/MSI: Getting rid of msi_controller, and other cleanups | expand

Commit Message

Marc Zyngier March 22, 2021, 6:46 p.m. UTC
The new 'no_msi' attribute solves the problem of advertising the lack
of MSI capability for host bridges that know for sure that there will
be no MSI for their end-points.

However, there is a whole class of host bridges that cannot know
whether MSIs will be provided or not, as they rely on other blocks
to provide the MSI functionnality, using MSI domains.  This is
the case for example on systems that use the ARM GIC architecture.

Introduce a new attribute ('msi_domain') indicating that implicit
dependency, and use this property to set the NO_MSI flag when
no MSI domain is found at probe time.

Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/pci/probe.c | 2 +-
 include/linux/pci.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

Comments

Robin Murphy March 23, 2021, 11:45 a.m. UTC | #1
On 2021-03-22 18:46, Marc Zyngier wrote:
> The new 'no_msi' attribute solves the problem of advertising the lack
> of MSI capability for host bridges that know for sure that there will
> be no MSI for their end-points.
> 
> However, there is a whole class of host bridges that cannot know
> whether MSIs will be provided or not, as they rely on other blocks
> to provide the MSI functionnality, using MSI domains.  This is
> the case for example on systems that use the ARM GIC architecture.
> 
> Introduce a new attribute ('msi_domain') indicating that implicit
> dependency, and use this property to set the NO_MSI flag when
> no MSI domain is found at probe time.
> 
> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>   drivers/pci/probe.c | 2 +-
>   include/linux/pci.h | 1 +
>   2 files changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> index 146bd85c037e..bac9f69a06a8 100644
> --- a/drivers/pci/probe.c
> +++ b/drivers/pci/probe.c
> @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
>   	device_enable_async_suspend(bus->bridge);
>   	pci_set_bus_of_node(bus);
>   	pci_set_bus_msi_domain(bus);
> -	if (bridge->no_msi)
> +	if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain))
>   		bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI;
>   
>   	if (!parent)
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index 48605cca82ae..d322d00db432 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -551,6 +551,7 @@ struct pci_host_bridge {
>   	unsigned int	preserve_config:1;	/* Preserve FW resource setup */
>   	unsigned int	size_windows:1;		/* Enable root bus sizing */
>   	unsigned int	no_msi:1;		/* Bridge has no MSI support */
> +	unsigned int	msi_domain:1;		/* Bridge wants MSI domain */

Aren't these really the same thing? Either way we're saying the bridge 
itself doesn't handle MSIs, it's just in one case we're effectively 
encoding a platform-specific assumption that an external domain won't be 
provided. I can't help wondering whether that distinction is really 
necessary...

Robin.

>   
>   	/* Resource alignment requirements */
>   	resource_size_t (*align_resource)(struct pci_dev *dev,
>
Marc Zyngier March 23, 2021, 6:09 p.m. UTC | #2
Hi Robin,

On Tue, 23 Mar 2021 11:45:02 +0000,
Robin Murphy <robin.murphy@arm.com> wrote:
> 
> On 2021-03-22 18:46, Marc Zyngier wrote:
> > The new 'no_msi' attribute solves the problem of advertising the lack
> > of MSI capability for host bridges that know for sure that there will
> > be no MSI for their end-points.
> > 
> > However, there is a whole class of host bridges that cannot know
> > whether MSIs will be provided or not, as they rely on other blocks
> > to provide the MSI functionnality, using MSI domains.  This is
> > the case for example on systems that use the ARM GIC architecture.
> > 
> > Introduce a new attribute ('msi_domain') indicating that implicit
> > dependency, and use this property to set the NO_MSI flag when
> > no MSI domain is found at probe time.
> > 
> > Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > ---
> >   drivers/pci/probe.c | 2 +-
> >   include/linux/pci.h | 1 +
> >   2 files changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > index 146bd85c037e..bac9f69a06a8 100644
> > --- a/drivers/pci/probe.c
> > +++ b/drivers/pci/probe.c
> > @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
> >   	device_enable_async_suspend(bus->bridge);
> >   	pci_set_bus_of_node(bus);
> >   	pci_set_bus_msi_domain(bus);
> > -	if (bridge->no_msi)
> > +	if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain))
> >   		bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI;
> >     	if (!parent)
> > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > index 48605cca82ae..d322d00db432 100644
> > --- a/include/linux/pci.h
> > +++ b/include/linux/pci.h
> > @@ -551,6 +551,7 @@ struct pci_host_bridge {
> >   	unsigned int	preserve_config:1;	/* Preserve FW resource setup */
> >   	unsigned int	size_windows:1;		/* Enable root bus sizing */
> >   	unsigned int	no_msi:1;		/* Bridge has no MSI support */
> > +	unsigned int	msi_domain:1;		/* Bridge wants MSI domain */
> 
> Aren't these really the same thing? Either way we're saying the bridge
> itself doesn't handle MSIs, it's just in one case we're effectively
> encoding a platform-specific assumption that an external domain won't
> be provided. I can't help wondering whether that distinction is really
> necessary...

There is a subtle difference: no_msi indicates that there is no way
*any* MSI can be dealt with whatsoever (maybe because the RC doesn't
forward the corresponding TLPs?). msi_domain says "no MSI unless...".

We could implement the former with the latter, but I have the feeling
that's not totally bullet proof. Happy to revisit this if you think it
really matters.

Thanks,

	M.
Robin Murphy March 23, 2021, 7:04 p.m. UTC | #3
On 2021-03-23 18:09, Marc Zyngier wrote:
> Hi Robin,
> 
> On Tue, 23 Mar 2021 11:45:02 +0000,
> Robin Murphy <robin.murphy@arm.com> wrote:
>>
>> On 2021-03-22 18:46, Marc Zyngier wrote:
>>> The new 'no_msi' attribute solves the problem of advertising the lack
>>> of MSI capability for host bridges that know for sure that there will
>>> be no MSI for their end-points.
>>>
>>> However, there is a whole class of host bridges that cannot know
>>> whether MSIs will be provided or not, as they rely on other blocks
>>> to provide the MSI functionnality, using MSI domains.  This is
>>> the case for example on systems that use the ARM GIC architecture.
>>>
>>> Introduce a new attribute ('msi_domain') indicating that implicit
>>> dependency, and use this property to set the NO_MSI flag when
>>> no MSI domain is found at probe time.
>>>
>>> Acked-by: Bjorn Helgaas <bhelgaas@google.com>
>>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>>> ---
>>>    drivers/pci/probe.c | 2 +-
>>>    include/linux/pci.h | 1 +
>>>    2 files changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
>>> index 146bd85c037e..bac9f69a06a8 100644
>>> --- a/drivers/pci/probe.c
>>> +++ b/drivers/pci/probe.c
>>> @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
>>>    	device_enable_async_suspend(bus->bridge);
>>>    	pci_set_bus_of_node(bus);
>>>    	pci_set_bus_msi_domain(bus);
>>> -	if (bridge->no_msi)
>>> +	if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain))
>>>    		bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI;
>>>      	if (!parent)
>>> diff --git a/include/linux/pci.h b/include/linux/pci.h
>>> index 48605cca82ae..d322d00db432 100644
>>> --- a/include/linux/pci.h
>>> +++ b/include/linux/pci.h
>>> @@ -551,6 +551,7 @@ struct pci_host_bridge {
>>>    	unsigned int	preserve_config:1;	/* Preserve FW resource setup */
>>>    	unsigned int	size_windows:1;		/* Enable root bus sizing */
>>>    	unsigned int	no_msi:1;		/* Bridge has no MSI support */
>>> +	unsigned int	msi_domain:1;		/* Bridge wants MSI domain */
>>
>> Aren't these really the same thing? Either way we're saying the bridge
>> itself doesn't handle MSIs, it's just in one case we're effectively
>> encoding a platform-specific assumption that an external domain won't
>> be provided. I can't help wondering whether that distinction is really
>> necessary...
> 
> There is a subtle difference: no_msi indicates that there is no way
> *any* MSI can be dealt with whatsoever (maybe because the RC doesn't
> forward the corresponding TLPs?). msi_domain says "no MSI unless...".

PCI says that MSIs are simply memory writes at the transaction level, so 
AFAIK unless the host bridge can't support DMA at all, it shouldn't be 
in a position to make any assumptions about what transactions mean what.

I suppose there could in theory be an issue in the other direction, 
where config space somehow didn't allow access to the MSI capability in 
the first place, but then we'd presumably just never detect any device 
as being MSI-capable in the first place, and it wouldn't matter either way.

> We could implement the former with the latter, but I have the feeling
> that's not totally bullet proof. Happy to revisit this if you think it
> really matters.

I would expect it to be a fairly safe assumption that a host bridge 
which "doesn't support MSIs" wouldn't have an msi-parent set, but I 
don't have a strong opinion either way - I just figured we could 
probably save a little bit of complexity here.

Cheers,
Robin.
Lorenzo Pieralisi March 24, 2021, 1:19 p.m. UTC | #4
On Tue, Mar 23, 2021 at 06:09:36PM +0000, Marc Zyngier wrote:
> Hi Robin,
> 
> On Tue, 23 Mar 2021 11:45:02 +0000,
> Robin Murphy <robin.murphy@arm.com> wrote:
> > 
> > On 2021-03-22 18:46, Marc Zyngier wrote:
> > > The new 'no_msi' attribute solves the problem of advertising the lack
> > > of MSI capability for host bridges that know for sure that there will
> > > be no MSI for their end-points.
> > > 
> > > However, there is a whole class of host bridges that cannot know
> > > whether MSIs will be provided or not, as they rely on other blocks
> > > to provide the MSI functionnality, using MSI domains.  This is
> > > the case for example on systems that use the ARM GIC architecture.
> > > 
> > > Introduce a new attribute ('msi_domain') indicating that implicit
> > > dependency, and use this property to set the NO_MSI flag when
> > > no MSI domain is found at probe time.
> > > 
> > > Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> > > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > > ---
> > >   drivers/pci/probe.c | 2 +-
> > >   include/linux/pci.h | 1 +
> > >   2 files changed, 2 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > > index 146bd85c037e..bac9f69a06a8 100644
> > > --- a/drivers/pci/probe.c
> > > +++ b/drivers/pci/probe.c
> > > @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
> > >   	device_enable_async_suspend(bus->bridge);
> > >   	pci_set_bus_of_node(bus);
> > >   	pci_set_bus_msi_domain(bus);
> > > -	if (bridge->no_msi)
> > > +	if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain))
> > >   		bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI;
> > >     	if (!parent)
> > > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > > index 48605cca82ae..d322d00db432 100644
> > > --- a/include/linux/pci.h
> > > +++ b/include/linux/pci.h
> > > @@ -551,6 +551,7 @@ struct pci_host_bridge {
> > >   	unsigned int	preserve_config:1;	/* Preserve FW resource setup */
> > >   	unsigned int	size_windows:1;		/* Enable root bus sizing */
> > >   	unsigned int	no_msi:1;		/* Bridge has no MSI support */
> > > +	unsigned int	msi_domain:1;		/* Bridge wants MSI domain */
> > 
> > Aren't these really the same thing? Either way we're saying the bridge
> > itself doesn't handle MSIs, it's just in one case we're effectively
> > encoding a platform-specific assumption that an external domain won't
> > be provided. I can't help wondering whether that distinction is really
> > necessary...
> 
> There is a subtle difference: no_msi indicates that there is no way
> *any* MSI can be dealt with whatsoever (maybe because the RC doesn't
> forward the corresponding TLPs?). msi_domain says "no MSI unless...".
> 
> We could implement the former with the latter, but I have the feeling
> that's not totally bullet proof. Happy to revisit this if you think it
> really matters.

IIUC msi_domain == 1 means: this host bridge needs an msi_domain to enable
MSIs, which in turn means that there are bridges that do _not_ require
an msi_domain to enable MSIs. I don't know how other arches handle the 
msi_domain pointer but I am asking whether making:

if (bridge->no_msi || !bus->dev.msi_domain))
	bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI;

is a possibility (removing the need for the msi_domain flag).

At least this looks more like an arch property than a host bridge
specific property (eg patch [13] pci_host_common_probe() may be used on
arches other than ARM where it is not necessary true that it requires an
msi_domain to enable MSIs).

I agree that's complicated to untangle - just asking if there is way
to simplify it.

Thanks,
Lorenzo

> Thanks,
> 
> 	M.
> 
> -- 
> Without deviation from the norm, progress is not possible.
Marc Zyngier March 24, 2021, 4:11 p.m. UTC | #5
On Wed, 24 Mar 2021 13:19:38 +0000,
Lorenzo Pieralisi <lorenzo.pieralisi@arm.com> wrote:
> 
> On Tue, Mar 23, 2021 at 06:09:36PM +0000, Marc Zyngier wrote:
> > Hi Robin,
> > 
> > On Tue, 23 Mar 2021 11:45:02 +0000,
> > Robin Murphy <robin.murphy@arm.com> wrote:
> > > 
> > > On 2021-03-22 18:46, Marc Zyngier wrote:
> > > > The new 'no_msi' attribute solves the problem of advertising the lack
> > > > of MSI capability for host bridges that know for sure that there will
> > > > be no MSI for their end-points.
> > > > 
> > > > However, there is a whole class of host bridges that cannot know
> > > > whether MSIs will be provided or not, as they rely on other blocks
> > > > to provide the MSI functionnality, using MSI domains.  This is
> > > > the case for example on systems that use the ARM GIC architecture.
> > > > 
> > > > Introduce a new attribute ('msi_domain') indicating that implicit
> > > > dependency, and use this property to set the NO_MSI flag when
> > > > no MSI domain is found at probe time.
> > > > 
> > > > Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> > > > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > > > ---
> > > >   drivers/pci/probe.c | 2 +-
> > > >   include/linux/pci.h | 1 +
> > > >   2 files changed, 2 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > > > index 146bd85c037e..bac9f69a06a8 100644
> > > > --- a/drivers/pci/probe.c
> > > > +++ b/drivers/pci/probe.c
> > > > @@ -925,7 +925,7 @@ static int pci_register_host_bridge(struct pci_host_bridge *bridge)
> > > >   	device_enable_async_suspend(bus->bridge);
> > > >   	pci_set_bus_of_node(bus);
> > > >   	pci_set_bus_msi_domain(bus);
> > > > -	if (bridge->no_msi)
> > > > +	if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain))
> > > >   		bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI;
> > > >     	if (!parent)
> > > > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > > > index 48605cca82ae..d322d00db432 100644
> > > > --- a/include/linux/pci.h
> > > > +++ b/include/linux/pci.h
> > > > @@ -551,6 +551,7 @@ struct pci_host_bridge {
> > > >   	unsigned int	preserve_config:1;	/* Preserve FW resource setup */
> > > >   	unsigned int	size_windows:1;		/* Enable root bus sizing */
> > > >   	unsigned int	no_msi:1;		/* Bridge has no MSI support */
> > > > +	unsigned int	msi_domain:1;		/* Bridge wants MSI domain */
> > > 
> > > Aren't these really the same thing? Either way we're saying the bridge
> > > itself doesn't handle MSIs, it's just in one case we're effectively
> > > encoding a platform-specific assumption that an external domain won't
> > > be provided. I can't help wondering whether that distinction is really
> > > necessary...
> > 
> > There is a subtle difference: no_msi indicates that there is no way
> > *any* MSI can be dealt with whatsoever (maybe because the RC doesn't
> > forward the corresponding TLPs?). msi_domain says "no MSI unless...".
> > 
> > We could implement the former with the latter, but I have the feeling
> > that's not totally bullet proof. Happy to revisit this if you think it
> > really matters.
> 
> IIUC msi_domain == 1 means: this host bridge needs an msi_domain to enable
> MSIs, which in turn means that there are bridges that do _not_ require
> an msi_domain to enable MSIs. I don't know how other arches handle the 
> msi_domain pointer but I am asking whether making:
> 
> if (bridge->no_msi || !bus->dev.msi_domain))
> 	bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI;
> 
> is a possibility (removing the need for the msi_domain flag).
> 
> At least this looks more like an arch property than a host bridge
> specific property (eg patch [13] pci_host_common_probe() may be used on
> arches other than ARM where it is not necessary true that it requires an
> msi_domain to enable MSIs).
> 
> I agree that's complicated to untangle - just asking if there is way
> to simplify it.

I tried to simplify that in the past (see the original discussion at
[1]), and tglx reported some breakages on systems that do not use MSI
domains, which is why we ended up with an explicit flag.

What I have done for now is to go with Robin's proposal of dropping
'no_msi' and rely on solely on 'msi_domain' to set
PCI_BUS_FLAGS_NO_MSI when no domain is found.

Note that if we indeed have a host bridge that uses
pci_host_common_probe() that doesn't use MSI domains, we may indeed
run into problems. I don't have a good way around that, unfortunately.

Thanks,

	M.

[1] https://lore.kernel.org/r/20201031140330.83768-1-linux@fw-web.de
diff mbox series

Patch

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 146bd85c037e..bac9f69a06a8 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -925,7 +925,7 @@  static int pci_register_host_bridge(struct pci_host_bridge *bridge)
 	device_enable_async_suspend(bus->bridge);
 	pci_set_bus_of_node(bus);
 	pci_set_bus_msi_domain(bus);
-	if (bridge->no_msi)
+	if (bridge->no_msi || (bridge->msi_domain && !bus->dev.msi_domain))
 		bus->bus_flags |= PCI_BUS_FLAGS_NO_MSI;
 
 	if (!parent)
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 48605cca82ae..d322d00db432 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -551,6 +551,7 @@  struct pci_host_bridge {
 	unsigned int	preserve_config:1;	/* Preserve FW resource setup */
 	unsigned int	size_windows:1;		/* Enable root bus sizing */
 	unsigned int	no_msi:1;		/* Bridge has no MSI support */
+	unsigned int	msi_domain:1;		/* Bridge wants MSI domain */
 
 	/* Resource alignment requirements */
 	resource_size_t (*align_resource)(struct pci_dev *dev,