| Message ID | 20230113171409.30470-26-Sergey.Semin@baikalelectronics.ru (mailing list archive) |
|---|---|
| State | Handled Elsewhere |
| Headers | show |
| Series | dmaengine: dw-edma: Add RP/EP local DMA controllers support | expand |
On Fri, Jan 13, 2023 at 08:14:07PM +0300, Serge Semin wrote: > The MSI target address requires to be reserved within the lowest 4GB > memory in order to support the PCIe peripherals with no 64-bit MSI TLPs > support. Since the allocation is done from the DMA-coherent memory let's > modify the allocation procedure to setting the coherent DMA-mask only and > avoiding the streaming DMA-mask modification. Thus at least the streaming > DMA operations would work with no artificial limitations. It will be > specifically useful for the eDMA-capable controllers so the corresponding > DMA-engine clients would map the DMA buffers with no need in the SWIOTLB > intervention for the buffers allocated above the 4GB memory region. > > While at it let's add a brief comment about the reason of having the MSI > target address allocated from the DMA-coherent memory limited with the 4GB > upper bound. > > Signed-off-by: Serge Semin <Sergey.Semin@baikalelectronics.ru> > Reviewed-by: Robin Murphy <robin.murphy@arm.com> > > --- > > Changelog v8: > - This is a new patch added on v8 stage of the series. > (@Robin, @Christoph) > --- > drivers/pci/controller/dwc/pcie-designware-host.c | 11 ++++++++++- > 1 file changed, 10 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c > index 3ab6ae3712c4..e10608af39b4 100644 > --- a/drivers/pci/controller/dwc/pcie-designware-host.c > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c > @@ -366,7 +366,16 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) > dw_chained_msi_isr, pp); > } > > - ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32)); > + /* > + * Even though the iMSI-RX Module supports 64-bit addresses some > + * peripheral PCIe devices may lack the 64-bit messages support. In > + * order not to miss MSI TLPs from those devices the MSI target address > + * has to be reserved within the lowest 4GB. > + * Note until there is a better alternative found the reservation is > + * done by allocating from the artificially limited DMA-coherent > + * memory. > + */ > + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)); We used to restrict both streaming and coherent DMA masks to 32 bits. Now we will only restrict coherent DMA to 32 bits. So in essence this change removes a restriction on the streaming DMA mask, right? And I guess bt1 is the only driver where this will make a difference (after the next patch) because no other dwc drivers set their own DMA masks? > if (ret) > dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n"); > > -- > 2.39.0 > >
On Fri, Jan 20, 2023 at 03:59:52PM -0600, Bjorn Helgaas wrote: > On Fri, Jan 13, 2023 at 08:14:07PM +0300, Serge Semin wrote: > > The MSI target address requires to be reserved within the lowest 4GB > > memory in order to support the PCIe peripherals with no 64-bit MSI TLPs > > support. Since the allocation is done from the DMA-coherent memory let's > > modify the allocation procedure to setting the coherent DMA-mask only and > > avoiding the streaming DMA-mask modification. Thus at least the streaming > > DMA operations would work with no artificial limitations. It will be > > specifically useful for the eDMA-capable controllers so the corresponding > > DMA-engine clients would map the DMA buffers with no need in the SWIOTLB > > intervention for the buffers allocated above the 4GB memory region. > > > > While at it let's add a brief comment about the reason of having the MSI > > target address allocated from the DMA-coherent memory limited with the 4GB > > upper bound. > > > > Signed-off-by: Serge Semin <Sergey.Semin@baikalelectronics.ru> > > Reviewed-by: Robin Murphy <robin.murphy@arm.com> > > > > --- > > > > Changelog v8: > > - This is a new patch added on v8 stage of the series. > > (@Robin, @Christoph) > > --- > > drivers/pci/controller/dwc/pcie-designware-host.c | 11 ++++++++++- > > 1 file changed, 10 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c > > index 3ab6ae3712c4..e10608af39b4 100644 > > --- a/drivers/pci/controller/dwc/pcie-designware-host.c > > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c > > @@ -366,7 +366,16 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) > > dw_chained_msi_isr, pp); > > } > > > > - ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32)); > > + /* > > + * Even though the iMSI-RX Module supports 64-bit addresses some > > + * peripheral PCIe devices may lack the 64-bit messages support. In > > + * order not to miss MSI TLPs from those devices the MSI target address > > + * has to be reserved within the lowest 4GB. > > + * Note until there is a better alternative found the reservation is > > + * done by allocating from the artificially limited DMA-coherent > > + * memory. > > + */ > > + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)); > > We used to restrict both streaming and coherent DMA masks to 32 bits. > Now we will only restrict coherent DMA to 32 bits. > > So in essence this change removes a restriction on the streaming DMA > mask, right? Right. > > And I guess bt1 is the only driver where this will make a difference > (after the next patch) because no other dwc drivers set their own DMA > masks? Right. But that's only because I am sure the Baikal-T1 PCIe eDMA engine is able to reach memory above 4GB limits. If you know any other DW PCIe RP/EP controller with eDMA-capability and which can work with the memory space wider than 4GB then you can update it' streaming DMA-mask too. Note lacking to do so doesn't mean the eDMA-engine driver won't work for them. The eDMA-engine will be forced to work with the lowest 4GB space to which the upper memory will be mapped via SWIOTLB. -Serge(y) > > > if (ret) > > dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n"); > > > > -- > > 2.39.0 > > > >
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c index 3ab6ae3712c4..e10608af39b4 100644 --- a/drivers/pci/controller/dwc/pcie-designware-host.c +++ b/drivers/pci/controller/dwc/pcie-designware-host.c @@ -366,7 +366,16 @@ static int dw_pcie_msi_host_init(struct dw_pcie_rp *pp) dw_chained_msi_isr, pp); } - ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32)); + /* + * Even though the iMSI-RX Module supports 64-bit addresses some + * peripheral PCIe devices may lack the 64-bit messages support. In + * order not to miss MSI TLPs from those devices the MSI target address + * has to be reserved within the lowest 4GB. + * Note until there is a better alternative found the reservation is + * done by allocating from the artificially limited DMA-coherent + * memory. + */ + ret = dma_set_coherent_mask(dev, DMA_BIT_MASK(32)); if (ret) dev_warn(dev, "Failed to set DMA mask to 32-bit. Devices with only 32-bit MSI support may not work properly\n");