[V2,00/40] genirq, irqchip: Convert ARM MSI handling to per device MSI domains

Message

Thomas Gleixner Nov. 21, 2022, 2:39 p.m. UTC

This builds on top of the per device MSI work which can be found here:

  https://lore.kernel.org/all/[e-mail]

and is available from git:

  git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git devmsi-v2-part3

This series converts the ARM MSI handling, especially the platform MSI
mechanics, to use per device MSI domains instead of the weird indirection
through platform MSI domains.

This also takes a different and less convoluted approach to the 'wire to
MSI' bridges by providing special MSI allocation/free functions which are
invoked from irq_create_fwspec_mapping() and irq_dispose_mapping() which
avoids the extra entry point msi_domain_populate_irqs() and its layering
violations.

The series has been tested on various systems (Ampere, Hisilicon HIP05,
ThunderX2, MVEBU MCBIN) but there is no full coverage yet.

Thanks to Marc, Mark and Robin for running this on their machines and
aiding with decoding the initial wreckage. Special thanks go to Marc for
patiently guiding me through the insanity of OF/ACPI related to this.

The series is also available from git:

  git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git devmsi-v2-arm

The work was done to validate that the design decisions for per device MSI
domains and the related data structures are up to the task to handle the
ARM oddities. The design held up, it was just necessary to extend some of
the data structures to handle the ARM specifics in a sane way, but there
was no fundamental shortcoming.

Thanks,

	tglx
---
 b/drivers/base/platform-msi.c                 |  354 +++-----------------------
 b/drivers/dma/mv_xor_v2.c                     |    8 
 b/drivers/dma/qcom/hidma.c                    |    6 
 b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c |    5 
 b/drivers/irqchip/Kconfig                     |    8 
 b/drivers/irqchip/Makefile                    |    4 
 b/drivers/irqchip/irq-gic-common.h            |    3 
 b/drivers/irqchip/irq-gic-msi-lib.c           |  136 +++++++++
 b/drivers/irqchip/irq-gic-msi-lib.h           |   27 +
 b/drivers/irqchip/irq-gic-v2m.c               |   93 ++----
 b/drivers/irqchip/irq-gic-v3-its-msi-parent.c |  207 +++++++++++++++
 b/drivers/irqchip/irq-gic-v3-its.c            |    4 
 b/drivers/irqchip/irq-gic-v3-mbi.c            |  137 ++--------
 b/drivers/irqchip/irq-gic-v3.c                |    6 
 b/drivers/irqchip/irq-imx-mu-msi.c            |   53 +--
 b/drivers/irqchip/irq-mbigen.c                |   98 ++-----
 b/drivers/irqchip/irq-mvebu-gicp.c            |   44 +--
 b/drivers/irqchip/irq-mvebu-icu.c             |  278 ++++++++------------
 b/drivers/irqchip/irq-mvebu-odmi.c            |   39 +-
 b/drivers/irqchip/irq-mvebu-sei.c             |   52 +--
 b/drivers/irqchip/irq-ti-sci-inta.c           |    2 
 b/drivers/mailbox/bcm-flexrm-mailbox.c        |    8 
 b/drivers/pci/msi/irqdomain.c                 |   30 ++
 b/drivers/perf/arm_smmuv3_pmu.c               |    4 
 b/include/linux/acpi_iort.h                   |    4 
 b/include/linux/irqdomain.h                   |   17 +
 b/include/linux/irqdomain_defs.h              |    2 
 b/include/linux/msi.h                         |   81 +----
 b/kernel/irq/irqdomain.c                      |   26 +
 b/kernel/irq/msi.c                            |  253 ++++++++++++------
 drivers/irqchip/irq-gic-v3-its-pci-msi.c      |  202 --------------
 drivers/irqchip/irq-gic-v3-its-platform-msi.c |  163 -----------
 32 files changed, 1021 insertions(+), 1333 deletions(-)

Comments

Frank Li Jan. 11, 2023, 7:29 p.m. UTC | #1

From: Frank Li <[e-mail]>

On Mon, Nov 21, 2022 at 03:39:28PM +0100, Thomas Gleixner wrote:
> The series is also available from git:
>
>   git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git devmsi-v2-arm
>
> The work was done to validate that the design decisions for per device MSI
> domains and the related data structures are up to the task to handle the
> ARM oddities. The design held up, it was just necessary to extend some of
> the data structures to handle the ARM specifics in a sane way, but there
> was no fundamental shortcoming.

I based on branch devmsi-arm, and run at ls1028a platform. And found 
Integrated Endpoint Root Complex and enetc can't work.

The reason is that pci_msi_domain_get_msi_rid can't get correct rid.

[    5.153596]  pci_msi_domain_get_msi_rid+0x94/0x110
[    5.158399]  its_pci_msi_prepare+0xac/0x180
[    5.162594]  __msi_domain_alloc_irqs+0x78/0x464
[    5.167138]  __msi_domain_alloc_locked+0x148/0x190
[    5.171942]  msi_domain_alloc_irqs_all_locked+0x54/0xb0
[    5.177184]  pci_msi_setup_msi_irqs+0x2c/0x4c
[    5.181551]  __pci_enable_msix_range+0x374/0x600
[    5.186182]  pci_alloc_irq_vectors_affinity+0xbc/0x13c
[    5.191336]  pci_alloc_irq_vectors+0x14/0x20
[    5.195618]  enetc_alloc_msix+0x40/0x2cc
[    5.199550]  enetc_pf_probe+0x788/0xea0
[    5.203395]  local_pci_probe+0x40/0x9c
[    5.207151]  pci_device_probe+0xac/0x20c


u32 pci_msi_domain_get_msi_rid(struct irq_domain *domain, struct pci_dev *pdev)
{
        struct device_node *of_node;
        u32 rid = pci_dev_id(pdev);

        pci_for_each_dma_alias(pdev, get_msi_id_cb, &rid);

        of_node = irq_domain_get_of_node(domain);
        rid = of_node ? of_msi_map_id(&pdev->dev, of_node, rid) :
                        iort_msi_map_id(&pdev->dev, rid);

}

supposed of_msi_map_id get map informaiton from device tree
msi-map = <0 &its 0x17 0xe>;


but when msi_create_device_irq_domain pre device msi domain, 
fwnode is NOT PCI host controller's device node.

bool msi_create_device_irq_domain()
{
	...
	if (bundle->info.flags & MSI_FLAG_USE_DEV_FWNODE)
                fwnode = dev->fwnode;
        else
                fwnode = fwnalloced = irq_domain_alloc_named_fwnode(bundle->name);
	...
}



[    4.601732]  msi_create_device_irq_domain+0x1bc/0x20c
[    4.606800]  pci_setup_msix_device_domain+0x98/0xd4
[    4.611691]  __pci_enable_msix_range+0x21c/0x600
[    4.616322]  pci_alloc_irq_vectors_affinity+0xbc/0x13c
[    4.621477]  pci_alloc_irq_vectors+0x14/0x20
[    4.625760]  enetc_alloc_msix+0x40/0x2cc
[    4.629693]  enetc_pf_probe+0x788/0xea0
[    4.633537]  local_pci_probe+0x40/0x9c
[    4.637294]  pci_device_probe+0xac/0x20c


I don't know how to correct fix this problem. Can you give some suggestion?

Frank