diff mbox series

[1/8] PCI/MSI: Enable and mask MSIX early

Message ID 20210721192650.106154171@linutronix.de (mailing list archive)
State Superseded
Delegated to: Bjorn Helgaas
Headers show
Series PCI/MSI, x86: Cure a couple of inconsistencies | expand

Commit Message

Thomas Gleixner July 21, 2021, 7:11 p.m. UTC
The ordering of MSI-X enable in hardware is disfunctional:

 1) MSI-X is disabled in the control register
 2) Various setup functions
 3) pci_msi_setup_msi_irqs() is invoked which ends up accessing
    the MSI-X table entries
 4) MSI-X is enabled and masked in the control register with the
    comment that enabling is required for some hardware to access
    the MSI-X table

#4 obviously contradicts #3. The history of this is an issue with the NIU
hardware. When #4 was introduced the table access actually happened in
msix_program_entries() which was invoked after enabling and masking MSI-X.

This was changed in commit d71d6432e105 ("PCI/MSI: Kill redundant call of
irq_set_msi_desc() for MSI-X interrupts") which removed the table write
from msix_program_entries().

Interestingly enough nobody noticed and either NIU still works or it did
not get any testing with a kernel 3.19 or later.

Nevertheless this is inconsistent and there is no reason why MSI-X can't be
enabled and masked in the control register early on, i.e. move #4 above to
#1. This preserves the NIU workaround and has no side effects on other
hardware.

Fixes: d71d6432e105 ("PCI/MSI: Kill redundant call of irq_set_msi_desc() for MSI-X interrupts")
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: David S. Miller <davem@davemloft.net>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: linux-pci@vger.kernel.org
---
 drivers/pci/msi.c |   28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

Comments

Raj, Ashok July 21, 2021, 9:38 p.m. UTC | #1
On Wed, Jul 21, 2021 at 09:11:27PM +0200, Thomas Gleixner wrote:
> The ordering of MSI-X enable in hardware is disfunctional:
> 
>  1) MSI-X is disabled in the control register
>  2) Various setup functions
>  3) pci_msi_setup_msi_irqs() is invoked which ends up accessing
>     the MSI-X table entries
>  4) MSI-X is enabled and masked in the control register with the
>     comment that enabling is required for some hardware to access
>     the MSI-X table
> 
> #4 obviously contradicts #3. The history of this is an issue with the NIU
> hardware. When #4 was introduced the table access actually happened in
> msix_program_entries() which was invoked after enabling and masking MSI-X.
> 
> This was changed in commit d71d6432e105 ("PCI/MSI: Kill redundant call of
> irq_set_msi_desc() for MSI-X interrupts") which removed the table write
> from msix_program_entries().
> 
> Interestingly enough nobody noticed and either NIU still works or it did
> not get any testing with a kernel 3.19 or later.
> 
> Nevertheless this is inconsistent and there is no reason why MSI-X can't be
> enabled and masked in the control register early on, i.e. move #4 above to

Does the above comment also apply to legacy MSI when it support per-vector
masking capability? Probably not interesting since without IR, we only give
1 vector to MSI. 

Reviewed-by: Ashok Raj <ashok.raj@intel.com>

> #1. This preserves the NIU workaround and has no side effects on other
> hardware.
> 
> Fixes: d71d6432e105 ("PCI/MSI: Kill redundant call of irq_set_msi_desc() for MSI-X interrupts")
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Cc: David S. Miller <davem@davemloft.net>
> Cc: Bjorn Helgaas <bhelgaas@google.com>
> Cc: linux-pci@vger.kernel.org
> ---
>  drivers/pci/msi.c |   28 +++++++++++++++-------------
>  1 file changed, 15 insertions(+), 13 deletions(-)
> 
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -772,18 +772,25 @@ static int msix_capability_init(struct p
>  	u16 control;
>  	void __iomem *base;
>  
> -	/* Ensure MSI-X is disabled while it is set up */
> -	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
> +	/*
> +	 * Some devices require MSI-X to be enabled before the MSI-X
> +	 * registers can be accessed.  Mask all the vectors to prevent
> +	 * interrupts coming in before they're fully set up.
> +	 */
> +	pci_msix_clear_and_set_ctrl(dev, 0, PCI_MSIX_FLAGS_MASKALL |
> +				    PCI_MSIX_FLAGS_ENABLE);
>  
>  	pci_read_config_word(dev, dev->msix_cap + PCI_MSIX_FLAGS, &control);
>  	/* Request & Map MSI-X table region */
>  	base = msix_map_region(dev, msix_table_size(control));
> -	if (!base)
> -		return -ENOMEM;
> +	if (!base) {
> +		ret = -ENOMEM;
> +		goto out_disable;
> +	}
>  
>  	ret = msix_setup_entries(dev, base, entries, nvec, affd);
>  	if (ret)
> -		return ret;
> +		goto out_disable;
>  
>  	ret = pci_msi_setup_msi_irqs(dev, nvec, PCI_CAP_ID_MSIX);
>  	if (ret)
> @@ -794,14 +801,6 @@ static int msix_capability_init(struct p
>  	if (ret)
>  		goto out_free;
>  
> -	/*
> -	 * Some devices require MSI-X to be enabled before we can touch the
> -	 * MSI-X registers.  We need to mask all the vectors to prevent
> -	 * interrupts coming in before they're fully set up.
> -	 */
> -	pci_msix_clear_and_set_ctrl(dev, 0,
> -				PCI_MSIX_FLAGS_MASKALL | PCI_MSIX_FLAGS_ENABLE);
> -
>  	msix_program_entries(dev, entries);
>  
>  	ret = populate_msi_sysfs(dev);
> @@ -836,6 +835,9 @@ static int msix_capability_init(struct p
>  out_free:
>  	free_msi_irqs(dev);
>  
> +out_disable:
> +	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
> +
>  	return ret;
>  }
>  
>
Thomas Gleixner July 21, 2021, 10:51 p.m. UTC | #2
On Wed, Jul 21 2021 at 14:38, Ashok Raj wrote:

> On Wed, Jul 21, 2021 at 09:11:27PM +0200, Thomas Gleixner wrote:
>> The ordering of MSI-X enable in hardware is disfunctional:
>> 
>>  1) MSI-X is disabled in the control register
>>  2) Various setup functions
>>  3) pci_msi_setup_msi_irqs() is invoked which ends up accessing
>>     the MSI-X table entries
>>  4) MSI-X is enabled and masked in the control register with the
>>     comment that enabling is required for some hardware to access
>>     the MSI-X table
>> 
>> #4 obviously contradicts #3. The history of this is an issue with the NIU
>> hardware. When #4 was introduced the table access actually happened in
>> msix_program_entries() which was invoked after enabling and masking MSI-X.
>> 
>> This was changed in commit d71d6432e105 ("PCI/MSI: Kill redundant call of
>> irq_set_msi_desc() for MSI-X interrupts") which removed the table write
>> from msix_program_entries().
>> 
>> Interestingly enough nobody noticed and either NIU still works or it did
>> not get any testing with a kernel 3.19 or later.
>> 
>> Nevertheless this is inconsistent and there is no reason why MSI-X can't be
>> enabled and masked in the control register early on, i.e. move #4 above to
>
> Does the above comment also apply to legacy MSI when it support per-vector
> masking capability? Probably not interesting since without IR, we only give
> 1 vector to MSI. 

No MSI is completely different as the MSI configuration is purely in PCI
config space while the MSI-X table is separately mapped.

Thanks,

        tglx
Bjorn Helgaas July 22, 2021, 9:43 p.m. UTC | #3
s/MSIX/MSI-X/ in subject

On Wed, Jul 21, 2021 at 09:11:27PM +0200, Thomas Gleixner wrote:
> The ordering of MSI-X enable in hardware is disfunctional:

s/disfunctional/dysfunctional/, isn't English wonderful ;)

>  1) MSI-X is disabled in the control register
>  2) Various setup functions
>  3) pci_msi_setup_msi_irqs() is invoked which ends up accessing
>     the MSI-X table entries
>  4) MSI-X is enabled and masked in the control register with the
>     comment that enabling is required for some hardware to access
>     the MSI-X table
> 
> #4 obviously contradicts #3. The history of this is an issue with the NIU

Annoyingly, if you "git rebase" and reword this commit log, it drops
this line and the one a few lines below because they start with "#".
Should be obvious, but took me a few iterations to see what was
happening.

> hardware. When #4 was introduced the table access actually happened in
> msix_program_entries() which was invoked after enabling and masking MSI-X.
> 
> This was changed in commit d71d6432e105 ("PCI/MSI: Kill redundant call of
> irq_set_msi_desc() for MSI-X interrupts") which removed the table write
> from msix_program_entries().
> 
> Interestingly enough nobody noticed and either NIU still works or it did
> not get any testing with a kernel 3.19 or later.
> 
> Nevertheless this is inconsistent and there is no reason why MSI-X can't be
> enabled and masked in the control register early on, i.e. move #4 above to
> #1. This preserves the NIU workaround and has no side effects on other
> hardware.
>
> Fixes: d71d6432e105 ("PCI/MSI: Kill redundant call of irq_set_msi_desc() for MSI-X interrupts")
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> Cc: David S. Miller <davem@davemloft.net>
> Cc: Bjorn Helgaas <bhelgaas@google.com>
> Cc: linux-pci@vger.kernel.org
> ---
>  drivers/pci/msi.c |   28 +++++++++++++++-------------
>  1 file changed, 15 insertions(+), 13 deletions(-)
> 
> --- a/drivers/pci/msi.c
> +++ b/drivers/pci/msi.c
> @@ -772,18 +772,25 @@ static int msix_capability_init(struct p
>  	u16 control;
>  	void __iomem *base;
>  
> -	/* Ensure MSI-X is disabled while it is set up */
> -	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
> +	/*
> +	 * Some devices require MSI-X to be enabled before the MSI-X
> +	 * registers can be accessed.  Mask all the vectors to prevent
> +	 * interrupts coming in before they're fully set up.
> +	 */
> +	pci_msix_clear_and_set_ctrl(dev, 0, PCI_MSIX_FLAGS_MASKALL |
> +				    PCI_MSIX_FLAGS_ENABLE);
>  
>  	pci_read_config_word(dev, dev->msix_cap + PCI_MSIX_FLAGS, &control);
>  	/* Request & Map MSI-X table region */
>  	base = msix_map_region(dev, msix_table_size(control));
> -	if (!base)
> -		return -ENOMEM;
> +	if (!base) {
> +		ret = -ENOMEM;
> +		goto out_disable;
> +	}
>  
>  	ret = msix_setup_entries(dev, base, entries, nvec, affd);
>  	if (ret)
> -		return ret;
> +		goto out_disable;
>  
>  	ret = pci_msi_setup_msi_irqs(dev, nvec, PCI_CAP_ID_MSIX);
>  	if (ret)
> @@ -794,14 +801,6 @@ static int msix_capability_init(struct p
>  	if (ret)
>  		goto out_free;
>  
> -	/*
> -	 * Some devices require MSI-X to be enabled before we can touch the
> -	 * MSI-X registers.  We need to mask all the vectors to prevent
> -	 * interrupts coming in before they're fully set up.
> -	 */
> -	pci_msix_clear_and_set_ctrl(dev, 0,
> -				PCI_MSIX_FLAGS_MASKALL | PCI_MSIX_FLAGS_ENABLE);
> -
>  	msix_program_entries(dev, entries);
>  
>  	ret = populate_msi_sysfs(dev);
> @@ -836,6 +835,9 @@ static int msix_capability_init(struct p
>  out_free:
>  	free_msi_irqs(dev);
>  
> +out_disable:
> +	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
> +
>  	return ret;
>  }
>  
>
Thomas Gleixner July 27, 2021, 8:33 p.m. UTC | #4
On Thu, Jul 22 2021 at 16:43, Bjorn Helgaas wrote:
> s/MSIX/MSI-X/ in subject

Sure.

> On Wed, Jul 21, 2021 at 09:11:27PM +0200, Thomas Gleixner wrote:
>> The ordering of MSI-X enable in hardware is disfunctional:
>
> s/disfunctional/dysfunctional/, isn't English wonderful ;)

Yes and I'm never going to master it.

>>  1) MSI-X is disabled in the control register
>>  2) Various setup functions
>>  3) pci_msi_setup_msi_irqs() is invoked which ends up accessing
>>     the MSI-X table entries
>>  4) MSI-X is enabled and masked in the control register with the
>>     comment that enabling is required for some hardware to access
>>     the MSI-X table
>> 
>> #4 obviously contradicts #3. The history of this is an issue with the NIU
>
> Annoyingly, if you "git rebase" and reword this commit log, it drops
> this line and the one a few lines below because they start with "#".
> Should be obvious, but took me a few iterations to see what was
> happening.

Cute.

Thanks,

        tglx
diff mbox series

Patch

--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -772,18 +772,25 @@  static int msix_capability_init(struct p
 	u16 control;
 	void __iomem *base;
 
-	/* Ensure MSI-X is disabled while it is set up */
-	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
+	/*
+	 * Some devices require MSI-X to be enabled before the MSI-X
+	 * registers can be accessed.  Mask all the vectors to prevent
+	 * interrupts coming in before they're fully set up.
+	 */
+	pci_msix_clear_and_set_ctrl(dev, 0, PCI_MSIX_FLAGS_MASKALL |
+				    PCI_MSIX_FLAGS_ENABLE);
 
 	pci_read_config_word(dev, dev->msix_cap + PCI_MSIX_FLAGS, &control);
 	/* Request & Map MSI-X table region */
 	base = msix_map_region(dev, msix_table_size(control));
-	if (!base)
-		return -ENOMEM;
+	if (!base) {
+		ret = -ENOMEM;
+		goto out_disable;
+	}
 
 	ret = msix_setup_entries(dev, base, entries, nvec, affd);
 	if (ret)
-		return ret;
+		goto out_disable;
 
 	ret = pci_msi_setup_msi_irqs(dev, nvec, PCI_CAP_ID_MSIX);
 	if (ret)
@@ -794,14 +801,6 @@  static int msix_capability_init(struct p
 	if (ret)
 		goto out_free;
 
-	/*
-	 * Some devices require MSI-X to be enabled before we can touch the
-	 * MSI-X registers.  We need to mask all the vectors to prevent
-	 * interrupts coming in before they're fully set up.
-	 */
-	pci_msix_clear_and_set_ctrl(dev, 0,
-				PCI_MSIX_FLAGS_MASKALL | PCI_MSIX_FLAGS_ENABLE);
-
 	msix_program_entries(dev, entries);
 
 	ret = populate_msi_sysfs(dev);
@@ -836,6 +835,9 @@  static int msix_capability_init(struct p
 out_free:
 	free_msi_irqs(dev);
 
+out_disable:
+	pci_msix_clear_and_set_ctrl(dev, PCI_MSIX_FLAGS_ENABLE, 0);
+
 	return ret;
 }