| Message ID | 20221022234024.87475-1-mw@semihalf.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | ARM: dts: armada-38x: Mark devices as dma-coherent | expand |
On Sun, Oct 23, 2022 at 01:40:24AM +0200, Marcin Wojtas wrote: > Armada 38x platforms marks all devices as coherent via > mvebu_hwcc_notifier(), whereas the standard way to determine > this is by of_dma_is_coherent(). Reflect the hardware > capabilities by adding 'dma-coherent' properties to the device tree. Hi Marcin Does this need to go to -rc for 6.0? The DMA issues being reported? If so, please add a Fixed: tag. Andrew
On Sun, Oct 23, 2022 at 05:04:01PM +0200, Andrew Lunn wrote: > On Sun, Oct 23, 2022 at 01:40:24AM +0200, Marcin Wojtas wrote: > > Armada 38x platforms marks all devices as coherent via > > mvebu_hwcc_notifier(), whereas the standard way to determine > > this is by of_dma_is_coherent(). Reflect the hardware > > capabilities by adding 'dma-coherent' properties to the device tree. > > Hi Marcin > > Does this need to go to -rc for 6.0? The DMA issues being reported? > If so, please add a Fixed: tag. Are we absolutely sure this makes sense? Looking at atch/arm/mach-mvebu/coherency.c, there are dependencies on stuff such as whether the kernel is in SMP mode or not (because the page tables need to be appropriately marked as shared for coherency with IO to work). We only enable the shared bit if we're in SMP mode because (a) its difficult to do at runtime due to TLB conflicts (requires switching the MMU off, rewriting the page tables and switching the MMU back on), and (b) setting the shared bit for CPUs that don't need it _can_ result in the CPUs basically bypassing their caches and thus kill system performance. So, if we have Armada 38x platforms that are operated in uniprocessor mode, this patch can cause havoc on such a setup. I would suggest utmost caution with this approach.
niedz., 23 paź 2022 o 18:21 Russell King (Oracle) <linux@armlinux.org.uk> napisał(a): > > On Sun, Oct 23, 2022 at 05:04:01PM +0200, Andrew Lunn wrote: > > On Sun, Oct 23, 2022 at 01:40:24AM +0200, Marcin Wojtas wrote: > > > Armada 38x platforms marks all devices as coherent via > > > mvebu_hwcc_notifier(), whereas the standard way to determine > > > this is by of_dma_is_coherent(). Reflect the hardware > > > capabilities by adding 'dma-coherent' properties to the device tree. > > > > Hi Marcin > > > > Does this need to go to -rc for 6.0? The DMA issues being reported? > > If so, please add a Fixed: tag. > > Are we absolutely sure this makes sense? > > Looking at atch/arm/mach-mvebu/coherency.c, there are dependencies > on stuff such as whether the kernel is in SMP mode or not (because > the page tables need to be appropriately marked as shared for > coherency with IO to work). We only enable the shared bit if we're > in SMP mode because (a) its difficult to do at runtime due to TLB > conflicts (requires switching the MMU off, rewriting the page tables > and switching the MMU back on), and (b) setting the shared bit for > CPUs that don't need it _can_ result in the CPUs basically bypassing > their caches and thus kill system performance. > > So, if we have Armada 38x platforms that are operated in uniprocessor > mode, this patch can cause havoc on such a setup. > > I would suggest utmost caution with this approach. > Sure. In such a case the description of 380 variant (single core) should remain untouched. We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about: - Don't change current behavior, i.e. perform a necessary kernel configuration in "arm,pl310-cache" driver, arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT - Satisfy of_dma_is_coherent() by adding `dma-coherent;` in armada-385.dtsi only (IMO this would describe HW properly) ? Best regards, Marcin
On Sun, 23 Oct 2022 23:30:34 +0200 Marcin Wojtas <mw@semihalf.com> wrote: > niedz., 23 paź 2022 o 18:21 Russell King (Oracle) > <linux@armlinux.org.uk> napisał(a): > > > > On Sun, Oct 23, 2022 at 05:04:01PM +0200, Andrew Lunn wrote: > > > On Sun, Oct 23, 2022 at 01:40:24AM +0200, Marcin Wojtas wrote: > > > > Armada 38x platforms marks all devices as coherent via > > > > mvebu_hwcc_notifier(), whereas the standard way to determine > > > > this is by of_dma_is_coherent(). Reflect the hardware > > > > capabilities by adding 'dma-coherent' properties to the device tree. > > > > > > Hi Marcin > > > > > > Does this need to go to -rc for 6.0? The DMA issues being reported? > > > If so, please add a Fixed: tag. > > > > Are we absolutely sure this makes sense? > > > > Looking at atch/arm/mach-mvebu/coherency.c, there are dependencies > > on stuff such as whether the kernel is in SMP mode or not (because > > the page tables need to be appropriately marked as shared for > > coherency with IO to work). We only enable the shared bit if we're > > in SMP mode because (a) its difficult to do at runtime due to TLB > > conflicts (requires switching the MMU off, rewriting the page tables > > and switching the MMU back on), and (b) setting the shared bit for > > CPUs that don't need it _can_ result in the CPUs basically bypassing > > their caches and thus kill system performance. > > > > So, if we have Armada 38x platforms that are operated in uniprocessor > > mode, this patch can cause havoc on such a setup. > > > > I would suggest utmost caution with this approach. > > > > Sure. In such a case the description of 380 variant (single core) > should remain untouched. > > We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about: > - Don't change current behavior, i.e. perform a necessary kernel > configuration in "arm,pl310-cache" driver, > arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT > - Satisfy of_dma_is_coherent() by adding `dma-coherent;` in > armada-385.dtsi only (IMO this would describe HW properly) > ? It will describe HW properly, but someone running older kernel compiled with no SMP support will see a performance drop. I wonder how many people do that. Marek
On Mon, Oct 24, 2022 at 08:51:02AM +0200, Marek Behún wrote: > > Sure. In such a case the description of 380 variant (single core) > > should remain untouched. > > > > We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about: > > - Don't change current behavior, i.e. perform a necessary kernel > > configuration in "arm,pl310-cache" driver, > > arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT > > - Satisfy of_dma_is_coherent() by adding `dma-coherent;` in > > armada-385.dtsi only (IMO this would describe HW properly) > > ? > > It will describe HW properly, but someone running older kernel compiled > with no SMP support will see a performance drop. I wonder how many > people do that. If the kernel is built without SMP support, the page table entries will not have the shared bit set, and the system will _not_ be DMA-coherent. Having DT mark devices as "dma-coherent" in this case will lead to data corruption, because the DMA API will believe them to be DMA-coherent when the page tables are not setup for that to work.
pon., 24 paź 2022 o 09:51 Russell King (Oracle) <linux@armlinux.org.uk> napisał(a): > > On Mon, Oct 24, 2022 at 08:51:02AM +0200, Marek Behún wrote: > > > Sure. In such a case the description of 380 variant (single core) > > > should remain untouched. > > > > > > We need to decide what to do with dual-CPU, i.e. Armada 385/388. How about: > > > - Don't change current behavior, i.e. perform a necessary kernel > > > configuration in "arm,pl310-cache" driver, > > > arch/arm/mach-mvebu/coherency.c + &coherencyfab:node in DT > > > - Satisfy of_dma_is_coherent() by adding `dma-coherent;` in > > > armada-385.dtsi only (IMO this would describe HW properly) > > > ? > > > > It will describe HW properly, but someone running older kernel compiled > > with no SMP support will see a performance drop. I wonder how many > > people do that. > > If the kernel is built without SMP support, the page table entries will > not have the shared bit set, and the system will _not_ be DMA-coherent. > Having DT mark devices as "dma-coherent" in this case will lead to data > corruption, because the DMA API will believe them to be DMA-coherent > when the page tables are not setup for that to work. > Thanks, for the explanation. Since we're heavily dependent on what happens in the kernel we boot, it will be easier to just drop this patch and keep using the DT as-is. Best regards, Marcin
diff --git a/arch/arm/boot/dts/armada-380.dtsi b/arch/arm/boot/dts/armada-380.dtsi index ce1dddb2269b..25d17550e2fc 100644 --- a/arch/arm/boot/dts/armada-380.dtsi +++ b/arch/arm/boot/dts/armada-380.dtsi @@ -38,6 +38,7 @@ pcie { compatible = "marvell,armada-370-pcie"; status = "disabled"; device_type = "pci"; + dma-coherent; #address-cells = <3>; #size-cells = <2>; diff --git a/arch/arm/boot/dts/armada-385.dtsi b/arch/arm/boot/dts/armada-385.dtsi index 83392b92dae2..6fb8c254cbdc 100644 --- a/arch/arm/boot/dts/armada-385.dtsi +++ b/arch/arm/boot/dts/armada-385.dtsi @@ -37,6 +37,7 @@ pciec: pcie { compatible = "marvell,armada-370-pcie"; status = "disabled"; device_type = "pci"; + dma-coherent; #address-cells = <3>; #size-cells = <2>; diff --git a/arch/arm/boot/dts/armada-38x.dtsi b/arch/arm/boot/dts/armada-38x.dtsi index 446861b6b17b..5801873dfcbe 100644 --- a/arch/arm/boot/dts/armada-38x.dtsi +++ b/arch/arm/boot/dts/armada-38x.dtsi @@ -102,6 +102,7 @@ internal-regs { #address-cells = <1>; #size-cells = <1>; ranges = <0 MBUS_ID(0xf0, 0x01) 0 0x100000>; + dma-coherent; sdramc: sdramc@1400 { compatible = "marvell,armada-xp-sdram-controller";
Armada 38x platforms marks all devices as coherent via mvebu_hwcc_notifier(), whereas the standard way to determine this is by of_dma_is_coherent(). Reflect the hardware capabilities by adding 'dma-coherent' properties to the device tree. Signed-off-by: Marcin Wojtas <mw@semihalf.com> --- arch/arm/boot/dts/armada-380.dtsi | 1 + arch/arm/boot/dts/armada-385.dtsi | 1 + arch/arm/boot/dts/armada-38x.dtsi | 1 + 3 files changed, 3 insertions(+)