mbox series

[00/18] ARM: Add minimal Raspberry Pi 4 support

Message ID 1563774880-8061-1-git-send-email-wahrenst@gmx.net (mailing list archive)
Headers show
Series ARM: Add minimal Raspberry Pi 4 support | expand

Message

Stefan Wahren July 22, 2019, 5:54 a.m. UTC
This series adds minimal support for the new Raspberry Pi 4, so we are able
to login via debug UART.

Patch 1-5:   Prepare platform and DTS for the new SoC BMC2711
Patch 6-10:  Enable support for emmc2 on BCM2711
Patch 11-12: Enable pinctrl for BCM2711
Patch 13-17: Add Raspberry Pi 4 DTS support
Patch 18:    Update MAINTAINERS

Unfortunately the Raspberry Pi Foundation didn't released a
peripheral documentation for the new SoC yet. So we only have a preliminary
datasheet [1] and reduced schematics [2].

Changes since RFC:
- change BCM2838 -> BCM2711 as discussed in RFC
- update MAINTAINERS accordingly
- drop "spi: bcm2835: enable shared interrupt support" from series
- squash all pinctrl-bcm2835 changes into one patch
- introduce SoC specific clock registration as suggested by Florian
- fix watchdog probing for Raspberry Pi 4
- convert brcm,bcm2835.txt to json-schema
- move VC4 node to bcm2835-common.dtsi
- fallback to legacy pull config for Raspberry Pi 4
- revert unintended change of mailbox in bcm283x.dtsi
- add reference for arm64

[1] - https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711/rpi_DATA_2711_1p0_preliminary.pdf
[2] - https://www.raspberrypi.org/documentation/hardware/raspberrypi/schematics/rpi_SCH_4b_4p0_reduced.pdf

Phil Elwell (1):
  ARM: bcm2835: DMA can only address 1GB

Stefan Wahren (17):
  ARM: bcm283x: Reduce register ranges for UART, SPI and I2C
  ARM: dts: bcm283x: Move BCM2835/6/7 specific to bcm2835-common.dtsi
  ARM: dts: bcm283x: Define MMC interfaces at board level
  ARM: dts: bcm283x: Define memory at board level
  dt-bindings: bcm2835-cprman: Add bcm2711 support
  clk: bcm2835: Introduce SoC specific clock registration
  clk: bcm2835: Add BCM2711_CLOCK_EMMC2 support
  dt-bindings: sdhci-iproc: Add brcm,bcm2711-emmc2
  mmc: sdhci-iproc: Add support for emmc2 of the BCM2711
  dt-bindings: pinctrl: bcm2835: Add brcm,bcm2711 compatible
  pinctrl: bcm2835: Add support for BCM2711 pull-up functionality
  dt-bindings: arm: Convert BCM2835 board/soc bindings to json-schema
  dt-bindings: arm: bcm2835: Add Raspberry Pi 4 to DT schema
  ARM: bcm2835: Add bcm2711 compatible string
  ARM: dts: Add minimal Raspberry Pi 4 support
  arm64: dts: broadcom: Add reference to RPi 4 B
  MAINTAINERS: Add BCM2711 to BCM2835 ARCH

 .../devicetree/bindings/arm/bcm/bcm2835.yaml       |  51 ++
 .../devicetree/bindings/arm/bcm/brcm,bcm2835.txt   |  67 ---
 .../bindings/clock/brcm,bcm2835-cprman.txt         |   4 +-
 .../devicetree/bindings/mmc/brcm,sdhci-iproc.txt   |   4 +-
 .../bindings/pinctrl/brcm,bcm2835-gpio.txt         |   1 +
 MAINTAINERS                                        |   3 +-
 arch/arm/boot/dts/Makefile                         |   1 +
 arch/arm/boot/dts/bcm2711-rpi-4-b.dts              | 120 ++++
 arch/arm/boot/dts/bcm2711.dtsi                     | 667 +++++++++++++++++++++
 arch/arm/boot/dts/bcm2835-common.dtsi              | 177 ++++++
 arch/arm/boot/dts/bcm2835-rpi-a-plus.dts           |  12 +
 arch/arm/boot/dts/bcm2835-rpi-a.dts                |  12 +
 arch/arm/boot/dts/bcm2835-rpi-b-plus.dts           |  12 +
 arch/arm/boot/dts/bcm2835-rpi-b-rev2.dts           |  12 +
 arch/arm/boot/dts/bcm2835-rpi-b.dts                |  12 +
 arch/arm/boot/dts/bcm2835-rpi-cm1-io1.dts          |   7 +
 arch/arm/boot/dts/bcm2835-rpi-cm1.dtsi             |   5 +
 arch/arm/boot/dts/bcm2835-rpi-zero-w.dts           |  12 +
 arch/arm/boot/dts/bcm2835-rpi-zero.dts             |  12 +
 arch/arm/boot/dts/bcm2835-rpi.dtsi                 |  18 -
 arch/arm/boot/dts/bcm2835.dtsi                     |   1 +
 arch/arm/boot/dts/bcm2836-rpi-2-b.dts              |   8 +
 arch/arm/boot/dts/bcm2836.dtsi                     |   1 +
 arch/arm/boot/dts/bcm2837-rpi-3-a-plus.dts         |   1 +
 arch/arm/boot/dts/bcm2837-rpi-3-b-plus.dts         |   1 +
 arch/arm/boot/dts/bcm2837-rpi-3-b.dts              |   1 +
 arch/arm/boot/dts/bcm2837-rpi-cm3-io3.dts          |   7 +
 arch/arm/boot/dts/bcm2837-rpi-cm3.dtsi             |   1 +
 arch/arm/boot/dts/bcm2837.dtsi                     |   1 +
 arch/arm/boot/dts/bcm283x.dtsi                     | 160 +----
 arch/arm/mach-bcm/board_bcm2835.c                  |   2 +
 arch/arm64/boot/dts/broadcom/Makefile              |   3 +-
 arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dts   |   2 +
 drivers/clk/bcm/clk-bcm2835.c                      | 117 +++-
 drivers/mmc/host/sdhci-iproc.c                     |   9 +
 drivers/pinctrl/bcm/pinctrl-bcm2835.c              | 105 +++-
 include/dt-bindings/clock/bcm2835.h                |   2 +
 37 files changed, 1365 insertions(+), 266 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/arm/bcm/bcm2835.yaml
 delete mode 100644 Documentation/devicetree/bindings/arm/bcm/brcm,bcm2835.txt
 create mode 100644 arch/arm/boot/dts/bcm2711-rpi-4-b.dts
 create mode 100644 arch/arm/boot/dts/bcm2711.dtsi
 create mode 100644 arch/arm/boot/dts/bcm2835-common.dtsi
 create mode 100644 arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dts

--
2.7.4

Comments

Matthias Brugger July 22, 2019, 11:26 a.m. UTC | #1
On 22/07/2019 07:54, Stefan Wahren wrote:
> This series adds minimal support for the new Raspberry Pi 4, so we are able
> to login via debug UART.
> 
> Patch 1-5:   Prepare platform and DTS for the new SoC BMC2711
> Patch 6-10:  Enable support for emmc2 on BCM2711
> Patch 11-12: Enable pinctrl for BCM2711
> Patch 13-17: Add Raspberry Pi 4 DTS support
> Patch 18:    Update MAINTAINERS

It seems that you only send patch 1-12 and in two batches. Can you please resend
or send the rest of the patches.

Regards,
Matthias

> 
> Unfortunately the Raspberry Pi Foundation didn't released a
> peripheral documentation for the new SoC yet. So we only have a preliminary
> datasheet [1] and reduced schematics [2].
> 
> Changes since RFC:
> - change BCM2838 -> BCM2711 as discussed in RFC
> - update MAINTAINERS accordingly
> - drop "spi: bcm2835: enable shared interrupt support" from series
> - squash all pinctrl-bcm2835 changes into one patch
> - introduce SoC specific clock registration as suggested by Florian
> - fix watchdog probing for Raspberry Pi 4
> - convert brcm,bcm2835.txt to json-schema
> - move VC4 node to bcm2835-common.dtsi
> - fallback to legacy pull config for Raspberry Pi 4
> - revert unintended change of mailbox in bcm283x.dtsi
> - add reference for arm64
> 
> [1] - https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711/rpi_DATA_2711_1p0_preliminary.pdf
> [2] - https://www.raspberrypi.org/documentation/hardware/raspberrypi/schematics/rpi_SCH_4b_4p0_reduced.pdf
> 
> Phil Elwell (1):
>   ARM: bcm2835: DMA can only address 1GB
> 
> Stefan Wahren (17):
>   ARM: bcm283x: Reduce register ranges for UART, SPI and I2C
>   ARM: dts: bcm283x: Move BCM2835/6/7 specific to bcm2835-common.dtsi
>   ARM: dts: bcm283x: Define MMC interfaces at board level
>   ARM: dts: bcm283x: Define memory at board level
>   dt-bindings: bcm2835-cprman: Add bcm2711 support
>   clk: bcm2835: Introduce SoC specific clock registration
>   clk: bcm2835: Add BCM2711_CLOCK_EMMC2 support
>   dt-bindings: sdhci-iproc: Add brcm,bcm2711-emmc2
>   mmc: sdhci-iproc: Add support for emmc2 of the BCM2711
>   dt-bindings: pinctrl: bcm2835: Add brcm,bcm2711 compatible
>   pinctrl: bcm2835: Add support for BCM2711 pull-up functionality
>   dt-bindings: arm: Convert BCM2835 board/soc bindings to json-schema
>   dt-bindings: arm: bcm2835: Add Raspberry Pi 4 to DT schema
>   ARM: bcm2835: Add bcm2711 compatible string
>   ARM: dts: Add minimal Raspberry Pi 4 support
>   arm64: dts: broadcom: Add reference to RPi 4 B
>   MAINTAINERS: Add BCM2711 to BCM2835 ARCH
> 
>  .../devicetree/bindings/arm/bcm/bcm2835.yaml       |  51 ++
>  .../devicetree/bindings/arm/bcm/brcm,bcm2835.txt   |  67 ---
>  .../bindings/clock/brcm,bcm2835-cprman.txt         |   4 +-
>  .../devicetree/bindings/mmc/brcm,sdhci-iproc.txt   |   4 +-
>  .../bindings/pinctrl/brcm,bcm2835-gpio.txt         |   1 +
>  MAINTAINERS                                        |   3 +-
>  arch/arm/boot/dts/Makefile                         |   1 +
>  arch/arm/boot/dts/bcm2711-rpi-4-b.dts              | 120 ++++
>  arch/arm/boot/dts/bcm2711.dtsi                     | 667 +++++++++++++++++++++
>  arch/arm/boot/dts/bcm2835-common.dtsi              | 177 ++++++
>  arch/arm/boot/dts/bcm2835-rpi-a-plus.dts           |  12 +
>  arch/arm/boot/dts/bcm2835-rpi-a.dts                |  12 +
>  arch/arm/boot/dts/bcm2835-rpi-b-plus.dts           |  12 +
>  arch/arm/boot/dts/bcm2835-rpi-b-rev2.dts           |  12 +
>  arch/arm/boot/dts/bcm2835-rpi-b.dts                |  12 +
>  arch/arm/boot/dts/bcm2835-rpi-cm1-io1.dts          |   7 +
>  arch/arm/boot/dts/bcm2835-rpi-cm1.dtsi             |   5 +
>  arch/arm/boot/dts/bcm2835-rpi-zero-w.dts           |  12 +
>  arch/arm/boot/dts/bcm2835-rpi-zero.dts             |  12 +
>  arch/arm/boot/dts/bcm2835-rpi.dtsi                 |  18 -
>  arch/arm/boot/dts/bcm2835.dtsi                     |   1 +
>  arch/arm/boot/dts/bcm2836-rpi-2-b.dts              |   8 +
>  arch/arm/boot/dts/bcm2836.dtsi                     |   1 +
>  arch/arm/boot/dts/bcm2837-rpi-3-a-plus.dts         |   1 +
>  arch/arm/boot/dts/bcm2837-rpi-3-b-plus.dts         |   1 +
>  arch/arm/boot/dts/bcm2837-rpi-3-b.dts              |   1 +
>  arch/arm/boot/dts/bcm2837-rpi-cm3-io3.dts          |   7 +
>  arch/arm/boot/dts/bcm2837-rpi-cm3.dtsi             |   1 +
>  arch/arm/boot/dts/bcm2837.dtsi                     |   1 +
>  arch/arm/boot/dts/bcm283x.dtsi                     | 160 +----
>  arch/arm/mach-bcm/board_bcm2835.c                  |   2 +
>  arch/arm64/boot/dts/broadcom/Makefile              |   3 +-
>  arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dts   |   2 +
>  drivers/clk/bcm/clk-bcm2835.c                      | 117 +++-
>  drivers/mmc/host/sdhci-iproc.c                     |   9 +
>  drivers/pinctrl/bcm/pinctrl-bcm2835.c              | 105 +++-
>  include/dt-bindings/clock/bcm2835.h                |   2 +
>  37 files changed, 1365 insertions(+), 266 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/arm/bcm/bcm2835.yaml
>  delete mode 100644 Documentation/devicetree/bindings/arm/bcm/brcm,bcm2835.txt
>  create mode 100644 arch/arm/boot/dts/bcm2711-rpi-4-b.dts
>  create mode 100644 arch/arm/boot/dts/bcm2711.dtsi
>  create mode 100644 arch/arm/boot/dts/bcm2835-common.dtsi
>  create mode 100644 arch/arm64/boot/dts/broadcom/bcm2711-rpi-4-b.dts
> 
> --
> 2.7.4
> 
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
Nicolas Saenz Julienne July 22, 2019, 1:18 p.m. UTC | #2
On Mon, 2019-07-22 at 07:54 +0200, Stefan Wahren wrote:
> This series adds minimal support for the new Raspberry Pi 4, so we are able
> to login via debug UART.
> 
> Patch 1-5:   Prepare platform and DTS for the new SoC BMC2711
> Patch 6-10:  Enable support for emmc2 on BCM2711
> Patch 11-12: Enable pinctrl for BCM2711
> Patch 13-17: Add Raspberry Pi 4 DTS support
> Patch 18:    Update MAINTAINERS
> 
> Unfortunately the Raspberry Pi Foundation didn't released a
> peripheral documentation for the new SoC yet. So we only have a preliminary
> datasheet [1] and reduced schematics [2].
> 
> Changes since RFC:
> - change BCM2838 -> BCM2711 as discussed in RFC
> - update MAINTAINERS accordingly
> - drop "spi: bcm2835: enable shared interrupt support" from series
> - squash all pinctrl-bcm2835 changes into one patch
> - introduce SoC specific clock registration as suggested by Florian
> - fix watchdog probing for Raspberry Pi 4
> - convert brcm,bcm2835.txt to json-schema
> - move VC4 node to bcm2835-common.dtsi
> - fallback to legacy pull config for Raspberry Pi 4
> - revert unintended change of mailbox in bcm283x.dtsi
> - add reference for arm64
> 
> [1] - 
> 
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711/rpi_DATA_2711_1p0_preliminary.pdf
> [2] - 
> 
https://www.raspberrypi.org/documentation/hardware/raspberrypi/schematics/rpi_SCH_4b_4p0_reduced.pdf
> 
> Phil Elwell (1):
>   ARM: bcm2835: DMA can only address 1GB
> 
> Stefan Wahren (17):
>   ARM: bcm283x: Reduce register ranges for UART, SPI and I2C
>   ARM: dts: bcm283x: Move BCM2835/6/7 specific to bcm2835-common.dtsi
>   ARM: dts: bcm283x: Define MMC interfaces at board level
>   ARM: dts: bcm283x: Define memory at board level
>   dt-bindings: bcm2835-cprman: Add bcm2711 support
>   clk: bcm2835: Introduce SoC specific clock registration
>   clk: bcm2835: Add BCM2711_CLOCK_EMMC2 support
>   dt-bindings: sdhci-iproc: Add brcm,bcm2711-emmc2
>   mmc: sdhci-iproc: Add support for emmc2 of the BCM2711
>   dt-bindings: pinctrl: bcm2835: Add brcm,bcm2711 compatible
>   pinctrl: bcm2835: Add support for BCM2711 pull-up functionality
>   dt-bindings: arm: Convert BCM2835 board/soc bindings to json-schema
>   dt-bindings: arm: bcm2835: Add Raspberry Pi 4 to DT schema
>   ARM: bcm2835: Add bcm2711 compatible string
>   ARM: dts: Add minimal Raspberry Pi 4 support
>   arm64: dts: broadcom: Add reference to RPi 4 B
>   MAINTAINERS: Add BCM2711 to BCM2835 ARCH

Hi Stefan,
I rebased your bcm2711-initial branch on top of v5.3-rc1 and got this:

[    2.307920] mmc1: new ultra high speed DDR50 SDHC card at address 59b4
[    2.314972] mmcblk1: mmc1:59b4 USD   15.0 GiB 
[    2.320240]  mmcblk1: p1 p2
[    2.324366] mmc0: new high speed SDIO card at address 0001
[    2.365315] EXT4-fs (mmcblk1p2): mounted filesystem with ordered data mode. Opts: (null)
[    2.373579] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
[    2.381220] devtmpfs: mounted
[    2.386618] Freeing unused kernel memory: 2048K
[    2.394373] Run /sbin/init as init process
[    2.644330] 8<--- cut here ---
[    2.647439] Unable to handle kernel paging request at virtual address bb000000
[    2.654795] pgd = (ptrval)
[    2.657540] [bb000000] *pgd=00000000
[    2.661172] Internal error: Oops: 2805 [#1] SMP ARM
[    2.666122] Modules linked in:
[    2.669225] CPU: 0 PID: 86 Comm: kworker/0:1H Not tainted 5.3.0-rc1-00020-g496d90bc72a9 #374
[    2.677792] Hardware name: BCM2835
[    2.681254] Workqueue: mmc_complete mmc_blk_mq_complete_work
[    2.687002] PC is at v7_dma_inv_range+0x3c/0x54
[    2.691602] LR is at __dma_page_dev_to_cpu+0x84/0x104
[    2.696728] pc : [<c031cfec>]    lr : [<c0317468>]    psr: 80000013
[    2.703089] sp : e8081e60  ip : ea059000  fp : 00000000
[    2.708390] r10: 00000002  r9 : 00000002  r8 : e9fb9000
[    2.713692] r7 : c1708cc4  r6 : c18cde80  r5 : 00001000  r4 : 00000000
[    2.720316] r3 : 0000003f  r2 : 00000040  r1 : bb001000  r0 : bb000000
[    2.726942] Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[    2.734185] Control: 10c5383d  Table: 0020406a  DAC: 00000051
[    2.740017] Process kworker/0:1H (pid: 86, stack limit = 0x(ptrval))
[    2.746466] Stack: (0xe8081e60 to 0xe8082000)
[    2.750891] 1e60: c031d084 ea004480 e928f300 e8053014 00000002 c0319350 c0f015d0 e9196c10
[    2.759197] 1e80: 00000002 c0317ac4 00000000 c124fc74 0000757d e97fb390 c0317a6c c1704c48
[    2.767503] 1ea0: e92a60c4 00000000 e97fb2b0 e9782800 e9783800 c0cce770 00000000 ea004480
[    2.775809] 1ec0: e928f300 e92a6008 e97fb200 c0cc8c68 e92a6008 e97fb200 c1704c48 e92a60c4
[    2.784115] 1ee0: 00000000 c0cc9040 00000000 e4380000 00000000 ada3354f 28989000 e92a60d8
[    2.792420] 1f00: e9286080 ea004280 ff7b0200 00000000 c1891c70 00000000 ea004280 c0360eb8
[    2.800726] 1f20: ffffe000 ea004280 c1703d00 e9286080 ea004280 e9286094 c1703d00 ea004298
[    2.809032] 1f40: ffffe000 00000008 ea004280 c0361bb4 ffffe000 c18915ac c124ea98 00000000
[    2.817338] 1f60: ffffe000 e91fc380 e92d3b00 00000000 e8080000 e9286080 c0361b70 e90bfea4
[    2.825643] 1f80: e91fc39c c0366dac 00000001 e92d3b00 c0366c64 00000000 00000000 00000000
[    2.833948] 1fa0: 00000000 00000000 00000000 c03010e8 00000000 00000000 00000000 00000000
[    2.842253] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.850559] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[    2.858872] [<c031cfec>] (v7_dma_inv_range) from [<c0317468>] (__dma_page_dev_to_cpu+0x84/0x104)
[    2.867797] [<c0317468>] (__dma_page_dev_to_cpu) from [<c0317ac4>] (arm_dma_unmap_sg+0x58/0x74)
[    2.876635] [<c0317ac4>] (arm_dma_unmap_sg) from [<c0cce770>] (sdhci_post_req+0x5c/0x8c)
[    2.884857] [<c0cce770>] (sdhci_post_req) from [<c0cc8c68>] (mmc_blk_mq_post_req+0x30/0xb4)
[    2.893340] [<c0cc8c68>] (mmc_blk_mq_post_req) from [<c0cc9040>] (mmc_blk_mq_complete_prev_req.part.4+0x174/0x238)
[    2.903855] [<c0cc9040>] (mmc_blk_mq_complete_prev_req.part.4) from [<c0360eb8>] (process_one_work+0x218/0x510)
[    2.914104] [<c0360eb8>] (process_one_work) from [<c0361bb4>] (worker_thread+0x44/0x5c0)
[    2.922323] [<c0361bb4>] (worker_thread) from [<c0366dac>] (kthread+0x148/0x150)
[    2.929836] [<c0366dac>] (kthread) from [<c03010e8>] (ret_from_fork+0x14/0x2c)
[    2.937168] Exception stack(0xe8081fb0 to 0xe8081ff8)
[    2.942296] 1fa0:                                     00000000 00000000 00000000 00000000
[    2.950601] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    2.958906] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[    2.965623] Code: e1110003 e1c11003 1e071f3e e1500001 (3e070f36) 
[    2.971830] ---[ end trace 918fd7aaa4c1e4cc ]---

I'm looking into it. Any ideas?

Regards,
Nicolas
Eric Anholt July 22, 2019, 6:01 p.m. UTC | #3
Stefan Wahren <wahrenst@gmx.net> writes:

> This series adds minimal support for the new Raspberry Pi 4, so we are able
> to login via debug UART.
>
> Patch 1-5:   Prepare platform and DTS for the new SoC BMC2711
> Patch 6-10:  Enable support for emmc2 on BCM2711
> Patch 11-12: Enable pinctrl for BCM2711
> Patch 13-17: Add Raspberry Pi 4 DTS support
> Patch 18:    Update MAINTAINERS
>
> Unfortunately the Raspberry Pi Foundation didn't released a
> peripheral documentation for the new SoC yet. So we only have a preliminary
> datasheet [1] and reduced schematics [2].
>
> Changes since RFC:
> - change BCM2838 -> BCM2711 as discussed in RFC
> - update MAINTAINERS accordingly
> - drop "spi: bcm2835: enable shared interrupt support" from series
> - squash all pinctrl-bcm2835 changes into one patch
> - introduce SoC specific clock registration as suggested by Florian
> - fix watchdog probing for Raspberry Pi 4
> - convert brcm,bcm2835.txt to json-schema
> - move VC4 node to bcm2835-common.dtsi
> - fallback to legacy pull config for Raspberry Pi 4
> - revert unintended change of mailbox in bcm283x.dtsi
> - add reference for arm64


I gave some comments on the bcm2835-common patch, but I think this is a
good start and would be worth merging to start bringing the board up,
even without a respin.

Series is:

Acked-by: Eric Anholt <eric@anholt.net>
Stefan Wahren July 22, 2019, 6:10 p.m. UTC | #4
Hi Nicolas,

[add Christoph]

Am 22.07.19 um 15:18 schrieb Nicolas Saenz Julienne:
> Hi Stefan,
> I rebased your bcm2711-initial branch on top of v5.3-rc1 and got this:
>
> [    2.307920] mmc1: new ultra high speed DDR50 SDHC card at address 59b4
> [    2.314972] mmcblk1: mmc1:59b4 USD   15.0 GiB
> [    2.320240]  mmcblk1: p1 p2
> [    2.324366] mmc0: new high speed SDIO card at address 0001
> [    2.365315] EXT4-fs (mmcblk1p2): mounted filesystem with ordered data mode. Opts: (null)
> [    2.373579] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.
> [    2.381220] devtmpfs: mounted
> [    2.386618] Freeing unused kernel memory: 2048K
> [    2.394373] Run /sbin/init as init process
> [    2.644330] 8<--- cut here ---
> [    2.647439] Unable to handle kernel paging request at virtual address bb000000
> [    2.654795] pgd = (ptrval)
> [    2.657540] [bb000000] *pgd=00000000
> [    2.661172] Internal error: Oops: 2805 [#1] SMP ARM
> [    2.666122] Modules linked in:
> [    2.669225] CPU: 0 PID: 86 Comm: kworker/0:1H Not tainted 5.3.0-rc1-00020-g496d90bc72a9 #374
> [    2.677792] Hardware name: BCM2835
> [    2.681254] Workqueue: mmc_complete mmc_blk_mq_complete_work
> [    2.687002] PC is at v7_dma_inv_range+0x3c/0x54
> [    2.691602] LR is at __dma_page_dev_to_cpu+0x84/0x104
> [    2.696728] pc : [<c031cfec>]    lr : [<c0317468>]    psr: 80000013
> [    2.703089] sp : e8081e60  ip : ea059000  fp : 00000000
> [    2.708390] r10: 00000002  r9 : 00000002  r8 : e9fb9000
> [    2.713692] r7 : c1708cc4  r6 : c18cde80  r5 : 00001000  r4 : 00000000
> [    2.720316] r3 : 0000003f  r2 : 00000040  r1 : bb001000  r0 : bb000000
> [    2.726942] Flags: Nzcv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
> [    2.734185] Control: 10c5383d  Table: 0020406a  DAC: 00000051
> [    2.740017] Process kworker/0:1H (pid: 86, stack limit = 0x(ptrval))
> [    2.746466] Stack: (0xe8081e60 to 0xe8082000)
> [    2.750891] 1e60: c031d084 ea004480 e928f300 e8053014 00000002 c0319350 c0f015d0 e9196c10
> [    2.759197] 1e80: 00000002 c0317ac4 00000000 c124fc74 0000757d e97fb390 c0317a6c c1704c48
> [    2.767503] 1ea0: e92a60c4 00000000 e97fb2b0 e9782800 e9783800 c0cce770 00000000 ea004480
> [    2.775809] 1ec0: e928f300 e92a6008 e97fb200 c0cc8c68 e92a6008 e97fb200 c1704c48 e92a60c4
> [    2.784115] 1ee0: 00000000 c0cc9040 00000000 e4380000 00000000 ada3354f 28989000 e92a60d8
> [    2.792420] 1f00: e9286080 ea004280 ff7b0200 00000000 c1891c70 00000000 ea004280 c0360eb8
> [    2.800726] 1f20: ffffe000 ea004280 c1703d00 e9286080 ea004280 e9286094 c1703d00 ea004298
> [    2.809032] 1f40: ffffe000 00000008 ea004280 c0361bb4 ffffe000 c18915ac c124ea98 00000000
> [    2.817338] 1f60: ffffe000 e91fc380 e92d3b00 00000000 e8080000 e9286080 c0361b70 e90bfea4
> [    2.825643] 1f80: e91fc39c c0366dac 00000001 e92d3b00 c0366c64 00000000 00000000 00000000
> [    2.833948] 1fa0: 00000000 00000000 00000000 c03010e8 00000000 00000000 00000000 00000000
> [    2.842253] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> [    2.850559] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
> [    2.858872] [<c031cfec>] (v7_dma_inv_range) from [<c0317468>] (__dma_page_dev_to_cpu+0x84/0x104)
> [    2.867797] [<c0317468>] (__dma_page_dev_to_cpu) from [<c0317ac4>] (arm_dma_unmap_sg+0x58/0x74)
> [    2.876635] [<c0317ac4>] (arm_dma_unmap_sg) from [<c0cce770>] (sdhci_post_req+0x5c/0x8c)
> [    2.884857] [<c0cce770>] (sdhci_post_req) from [<c0cc8c68>] (mmc_blk_mq_post_req+0x30/0xb4)
> [    2.893340] [<c0cc8c68>] (mmc_blk_mq_post_req) from [<c0cc9040>] (mmc_blk_mq_complete_prev_req.part.4+0x174/0x238)
> [    2.903855] [<c0cc9040>] (mmc_blk_mq_complete_prev_req.part.4) from [<c0360eb8>] (process_one_work+0x218/0x510)
> [    2.914104] [<c0360eb8>] (process_one_work) from [<c0361bb4>] (worker_thread+0x44/0x5c0)
> [    2.922323] [<c0361bb4>] (worker_thread) from [<c0366dac>] (kthread+0x148/0x150)
> [    2.929836] [<c0366dac>] (kthread) from [<c03010e8>] (ret_from_fork+0x14/0x2c)
> [    2.937168] Exception stack(0xe8081fb0 to 0xe8081ff8)
> [    2.942296] 1fa0:                                     00000000 00000000 00000000 00000000
> [    2.950601] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
> [    2.958906] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> [    2.965623] Code: e1110003 e1c11003 1e071f3e e1500001 (3e070f36)
> [    2.971830] ---[ end trace 918fd7aaa4c1e4cc ]---
>
> I'm looking into it. Any ideas?

i rebased this series also and got this only on the RPi 4.

After reverting the following:

79a986721de dma-mapping: remove dma_max_pfn
7559d612dff0 mmc: core: let the dma map ops handle bouncing

This crash disappear, but wifi seems to be still broken.

Would be nice, if you can investigate further.

Regards
Stefan

>
> Regards,
> Nicolas
>
Christoph Hellwig July 23, 2019, 9:34 a.m. UTC | #5
On Mon, Jul 22, 2019 at 08:10:17PM +0200, Stefan Wahren wrote:
> i rebased this series also and got this only on the RPi 4.
> 
> After reverting the following:
> 
> 79a986721de dma-mapping: remove dma_max_pfn
> 7559d612dff0 mmc: core: let the dma map ops handle bouncing
> 
> This crash disappear, but wifi seems to be still broken.
> 
> Would be nice, if you can investigate further.

That means dma addressing on this system doesn't just work for some
memory, and the mmc bounce buffering was papering over that just for
mmc.  Do you have highmem on this system?  

You might want to try this series, which has been submitted upstream:

http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/arm-swiotlb
Nicolas Saenz Julienne July 23, 2019, 1:32 p.m. UTC | #6
On Tue, 2019-07-23 at 11:34 +0200, Christoph Hellwig wrote:
> On Mon, Jul 22, 2019 at 08:10:17PM +0200, Stefan Wahren wrote:
> > i rebased this series also and got this only on the RPi 4.
> > 
> > After reverting the following:
> > 
> > 79a986721de dma-mapping: remove dma_max_pfn
> > 7559d612dff0 mmc: core: let the dma map ops handle bouncing
> > 
> > This crash disappear, but wifi seems to be still broken.
> > 
> > Would be nice, if you can investigate further.
> 
> That means dma addressing on this system doesn't just work for some
> memory, and the mmc bounce buffering was papering over that just for
> mmc.  Do you have highmem on this system? 
> 
> You might want to try this series, which has been submitted upstream:
> 
> http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/arm-swiotlb

Hi Christoph,
I tried your series on top of Stefan's, it has no effect. I guess it's no
surprise as with mult_v7_defconfig, you get SWIOTLB=n & LPAE=n.

FYI DMA addressing constraints for RPi4 are the following: devices can only
access the first GB of ram even though the board might have up to 4GB of ram.
The DMA addresses are aliased with a 0xc0000000 offset. So 0x00000000 phys is
aliased to 0xc0000000 in DMA. This is the same as for an RFC you commented last
week trying to fix similar issues for arm64.

You state in "arm: use swiotlb for bounce buffer on LPAE configs" that "The DMA
API requires that 32-bit DMA masks are always supported". If I understand it
correctly this device breaks that assumption. Which implies we need a bounce
buffer system in place for any straming DMA user.

It seems we're unable to use dma-direct/swiotlb, so I enabled arm's dmabounce
on all devices hooked into RPi's limited interconnect, which fixes this issue.
Any thoughts on this?

diff --git a/arch/arm/mach-bcm/Kconfig b/arch/arm/mach-bcm/Kconfig
index 5e5f1fabc3d4..3db8deed83a6 100644
--- a/arch/arm/mach-bcm/Kconfig
+++ b/arch/arm/mach-bcm/Kconfig
@@ -168,6 +168,7 @@ config ARCH_BCM2835
        select PINCTRL
        select PINCTRL_BCM2835
        select MFD_CORE
+       select DMABOUNCE
        help
          This enables support for the Broadcom BCM2835 and BCM2836 SoCs.
          This SoC is used in the Raspberry Pi and Roku 2 devices.
diff --git a/arch/arm/mach-bcm/board_bcm2835.c b/arch/arm/mach-bcm/board_bcm2835.c
index c09cf25596af..be788849c4bb 100644
--- a/arch/arm/mach-bcm/board_bcm2835.c
+++ b/arch/arm/mach-bcm/board_bcm2835.c
@@ -3,6 +3,8 @@
  * Copyright (C) 2010 Broadcom
  */
 
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
 #include <linux/init.h>
 #include <linux/irqchip.h>
 #include <linux/of_address.h>
@@ -24,8 +26,37 @@ static const char * const bcm2835_compat[] = {
        NULL
 };
 
+static int bcm2835_needs_bounce(struct device *dev, dma_addr_t dma_addr, size_t size)
+{
+       /*
+        * The accepted dma addresses are [0xc0000000, 0xffffffff] which map to
+        * ram's [0x00000000, 0x3fffffff].
+        */
+       return dma_addr < 3ULL * SZ_1G;
+}
+
+/*
+ * Setup DMA mask to 1GB on devices hanging from soc interconnect
+ */
+static int bcm2835_platform_notify(struct device *dev)
+{
+       if (dev->parent && !strcmp("soc", dev_name(dev->parent))) {
+               dev->dma_mask = &dev->coherent_dma_mask;
+               dev->coherent_dma_mask = DMA_BIT_MASK(30); /* 1GB */
+               dmabounce_register_dev(dev, 2048, 4096, bcm2835_needs_bounce);
+       }
+
+       return 0;
+}
+
+void __init bcm2835_init_early(void)
+{
+       platform_notify = bcm2835_platform_notify;
+}
+
 DT_MACHINE_START(BCM2835, "BCM2835")
        .dma_zone_size  = SZ_1G,
        .dt_compat = bcm2835_compat,
        .smp = smp_ops(bcm2836_smp_ops),
+       .init_early = bcm2835_init_early,
 MACHINE_END

 Regards,
 Nicolas
Christoph Hellwig July 23, 2019, 2:33 p.m. UTC | #7
On Tue, Jul 23, 2019 at 03:32:11PM +0200, Nicolas Saenz Julienne wrote:
> You state in "arm: use swiotlb for bounce buffer on LPAE configs" that "The DMA
> API requires that 32-bit DMA masks are always supported". If I understand it
> correctly this device breaks that assumption. Which implies we need a bounce
> buffer system in place for any straming DMA user.

Yes, you do.

> It seems we're unable to use dma-direct/swiotlb, so I enabled arm's dmabounce
> on all devices hooked into RPi's limited interconnect, which fixes this issue.
> Any thoughts on this?

There is no reason swiotlb could not handle the case, but at least for
now dmabounce seems like the better option given that it is well
integrated into the arm code.

Your patch looks good to me.
Stefan Wahren July 23, 2019, 4:26 p.m. UTC | #8
Hi Nicolas,

thanks for your work, but i'm a little bit sceptical about these
changes. So here some thoughts.

Am 23.07.19 um 15:32 schrieb Nicolas Saenz Julienne:
> On Tue, 2019-07-23 at 11:34 +0200, Christoph Hellwig wrote:
>> On Mon, Jul 22, 2019 at 08:10:17PM +0200, Stefan Wahren wrote:
>>> i rebased this series also and got this only on the RPi 4.
>>>
>>> After reverting the following:
>>>
>>> 79a986721de dma-mapping: remove dma_max_pfn
>>> 7559d612dff0 mmc: core: let the dma map ops handle bouncing
>>>
>>> This crash disappear, but wifi seems to be still broken.
>>>
>>> Would be nice, if you can investigate further.
>> That means dma addressing on this system doesn't just work for some
>> memory, and the mmc bounce buffering was papering over that just for
>> mmc.  Do you have highmem on this system?
>>
>> You might want to try this series, which has been submitted upstream:
>>
>> http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/arm-swiotlb
> Hi Christoph,
> I tried your series on top of Stefan's, it has no effect. I guess it's no
> surprise as with mult_v7_defconfig, you get SWIOTLB=n & LPAE=n.
>
> FYI DMA addressing constraints for RPi4 are the following: devices can only
> access the first GB of ram even though the board might have up to 4GB of ram.
> The DMA addresses are aliased with a 0xc0000000 offset. So 0x00000000 phys is
> aliased to 0xc0000000 in DMA. This is the same as for an RFC you commented last
> week trying to fix similar issues for arm64.
>
> You state in "arm: use swiotlb for bounce buffer on LPAE configs" that "The DMA
> API requires that 32-bit DMA masks are always supported". If I understand it
> correctly this device breaks that assumption. Which implies we need a bounce
> buffer system in place for any straming DMA user.
>
> It seems we're unable to use dma-direct/swiotlb, so I enabled arm's dmabounce
> on all devices hooked into RPi's limited interconnect, which fixes this issue.
Does it fix the wifi issue too?
> Any thoughts on this?
>
> diff --git a/arch/arm/mach-bcm/Kconfig b/arch/arm/mach-bcm/Kconfig
> index 5e5f1fabc3d4..3db8deed83a6 100644
> --- a/arch/arm/mach-bcm/Kconfig
> +++ b/arch/arm/mach-bcm/Kconfig
> @@ -168,6 +168,7 @@ config ARCH_BCM2835
>         select PINCTRL
>         select PINCTRL_BCM2835
>         select MFD_CORE
> +       select DMABOUNCE
>         help
>           This enables support for the Broadcom BCM2835 and BCM2836 SoCs.
>           This SoC is used in the Raspberry Pi and Roku 2 devices.
> diff --git a/arch/arm/mach-bcm/board_bcm2835.c b/arch/arm/mach-bcm/board_bcm2835.c
> index c09cf25596af..be788849c4bb 100644
> --- a/arch/arm/mach-bcm/board_bcm2835.c
> +++ b/arch/arm/mach-bcm/board_bcm2835.c
> @@ -3,6 +3,8 @@
>   * Copyright (C) 2010 Broadcom
>   */
>
> +#include <linux/device.h>
> +#include <linux/dma-mapping.h>
>  #include <linux/init.h>
>  #include <linux/irqchip.h>
>  #include <linux/of_address.h>
> @@ -24,8 +26,37 @@ static const char * const bcm2835_compat[] = {
>         NULL
>  };
>
> +static int bcm2835_needs_bounce(struct device *dev, dma_addr_t dma_addr, size_t size)
> +{
> +       /*
> +        * The accepted dma addresses are [0xc0000000, 0xffffffff] which map to
> +        * ram's [0x00000000, 0x3fffffff].
> +        */
> +       return dma_addr < 3ULL * SZ_1G;
> +}
> +
> +/*
> + * Setup DMA mask to 1GB on devices hanging from soc interconnect
> + */
> +static int bcm2835_platform_notify(struct device *dev)
> +{
> +       if (dev->parent && !strcmp("soc", dev_name(dev->parent))) {
> +               dev->dma_mask = &dev->coherent_dma_mask;
> +               dev->coherent_dma_mask = DMA_BIT_MASK(30); /* 1GB */
Shouldn't this come from the device tree?
> +               dmabounce_register_dev(dev, 2048, 4096, bcm2835_needs_bounce);
> +       }
> +
> +       return 0;
> +}
> +
> +void __init bcm2835_init_early(void)
> +{
> +       platform_notify = bcm2835_platform_notify;
> +}
> +
>  DT_MACHINE_START(BCM2835, "BCM2835")
>         .dma_zone_size  = SZ_1G,
>         .dt_compat = bcm2835_compat,
>         .smp = smp_ops(bcm2836_smp_ops),
> +       .init_early = bcm2835_init_early,

The sum of all these changes make me think, that we should start a new
board for BCM2711 instead of extending BCM2835.

Best regards
Stefan Wahren

>  MACHINE_END
>
>  Regards,
>  Nicolas
>
Nicolas Saenz Julienne July 23, 2019, 5:33 p.m. UTC | #9
On Tue, 2019-07-23 at 18:26 +0200, Stefan Wahren wrote:
> Hi Nicolas,
> 
> thanks for your work, but i'm a little bit sceptical about these
> changes. So here some thoughts.
> 
> Am 23.07.19 um 15:32 schrieb Nicolas Saenz Julienne:
> > On Tue, 2019-07-23 at 11:34 +0200, Christoph Hellwig wrote:
> > > On Mon, Jul 22, 2019 at 08:10:17PM +0200, Stefan Wahren wrote:
> > > > i rebased this series also and got this only on the RPi 4.
> > > > 
> > > > After reverting the following:
> > > > 
> > > > 79a986721de dma-mapping: remove dma_max_pfn
> > > > 7559d612dff0 mmc: core: let the dma map ops handle bouncing
> > > > 
> > > > This crash disappear, but wifi seems to be still broken.
> > > > 
> > > > Would be nice, if you can investigate further.
> > > That means dma addressing on this system doesn't just work for some
> > > memory, and the mmc bounce buffering was papering over that just for
> > > mmc.  Do you have highmem on this system?
> > > 
> > > You might want to try this series, which has been submitted upstream:
> > > 
> > > 
http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/arm-swiotlb
> > Hi Christoph,
> > I tried your series on top of Stefan's, it has no effect. I guess it's no
> > surprise as with mult_v7_defconfig, you get SWIOTLB=n & LPAE=n.
> > 
> > FYI DMA addressing constraints for RPi4 are the following: devices can only
> > access the first GB of ram even though the board might have up to 4GB of
> > ram.
> > The DMA addresses are aliased with a 0xc0000000 offset. So 0x00000000 phys
> > is
> > aliased to 0xc0000000 in DMA. This is the same as for an RFC you commented
> > last
> > week trying to fix similar issues for arm64.
> > 
> > You state in "arm: use swiotlb for bounce buffer on LPAE configs" that "The
> > DMA
> > API requires that 32-bit DMA masks are always supported". If I understand it
> > correctly this device breaks that assumption. Which implies we need a bounce
> > buffer system in place for any straming DMA user.
> > 
> > It seems we're unable to use dma-direct/swiotlb, so I enabled arm's
> > dmabounce
> > on all devices hooked into RPi's limited interconnect, which fixes this
> > issue.
> Does it fix the wifi issue too?

Well it works as long as I revert this: 901bb98918 ("nl80211: require and
validate vendor command policy"). Which has nothing to do with DMA anyways.

Was this the issue you where seeing?

[    4.969679] WARNING: CPU: 2 PID: 21 at net/wireless/core.c:868 wiphy_register+0x8e8/0xbdc [cfg80211]
[...]
[    4.969974] ieee80211 phy0: brcmf_cfg80211_attach: Could not register wiphy device (-22)

> > Any thoughts on this?
> > 
> > diff --git a/arch/arm/mach-bcm/Kconfig b/arch/arm/mach-bcm/Kconfig
> > index 5e5f1fabc3d4..3db8deed83a6 100644
> > --- a/arch/arm/mach-bcm/Kconfig
> > +++ b/arch/arm/mach-bcm/Kconfig
> > @@ -168,6 +168,7 @@ config ARCH_BCM2835
> >         select PINCTRL
> >         select PINCTRL_BCM2835
> >         select MFD_CORE
> > +       select DMABOUNCE
> >         help
> >           This enables support for the Broadcom BCM2835 and BCM2836 SoCs.
> >           This SoC is used in the Raspberry Pi and Roku 2 devices.
> > diff --git a/arch/arm/mach-bcm/board_bcm2835.c b/arch/arm/mach-
> > bcm/board_bcm2835.c
> > index c09cf25596af..be788849c4bb 100644
> > --- a/arch/arm/mach-bcm/board_bcm2835.c
> > +++ b/arch/arm/mach-bcm/board_bcm2835.c
> > @@ -3,6 +3,8 @@
> >   * Copyright (C) 2010 Broadcom
> >   */
> > 
> > +#include <linux/device.h>
> > +#include <linux/dma-mapping.h>
> >  #include <linux/init.h>
> >  #include <linux/irqchip.h>
> >  #include <linux/of_address.h>
> > @@ -24,8 +26,37 @@ static const char * const bcm2835_compat[] = {
> >         NULL
> >  };
> > 
> > +static int bcm2835_needs_bounce(struct device *dev, dma_addr_t dma_addr,
> > size_t size)
> > +{
> > +       /*
> > +        * The accepted dma addresses are [0xc0000000, 0xffffffff] which map
> > to
> > +        * ram's [0x00000000, 0x3fffffff].
> > +        */
> > +       return dma_addr < 3ULL * SZ_1G;
> > +}
> > +
> > +/*
> > + * Setup DMA mask to 1GB on devices hanging from soc interconnect
> > + */
> > +static int bcm2835_platform_notify(struct device *dev)
> > +{
> > +       if (dev->parent && !strcmp("soc", dev_name(dev->parent))) {
> > +               dev->dma_mask = &dev->coherent_dma_mask;
> > +               dev->coherent_dma_mask = DMA_BIT_MASK(30); /* 1GB */
> Shouldn't this come from the device tree?

Yes, actually I could use the 'dma-ranges' parsing code I suggested on the
arm64 RFC. The same goes with 'dma_zone_size = SZ_1G', it ideally should be
calculated based on the device-tree.

The way I see it I'm not sure it's worth the effort, in arm64 we have no choice
as there are no board files. But here we seem to be the only ones with this
specific DMA addressing constraint, so fixing it in arm/common doesn't seem
like it's going to benefit anyone else. Let's see how the arm arch maintainers
react though.

There is one catch though. I missed it earlier as I was excited to see the
board boot, but some devices are failing to set their DMA masks:

[    1.989576] dwc2 fe980000.usb: can't set coherent DMA mask: -5

It seems that other users of dmabounce also implement their own
dma_supported(). I have to look into it.

> > +               dmabounce_register_dev(dev, 2048, 4096,
> > bcm2835_needs_bounce);
> > +       }
> > +
> > +       return 0;
> > +}
> > +
> > +void __init bcm2835_init_early(void)
> > +{
> > +       platform_notify = bcm2835_platform_notify;
> > +}
> > +
> >  DT_MACHINE_START(BCM2835, "BCM2835")
> >         .dma_zone_size  = SZ_1G,
> >         .dt_compat = bcm2835_compat,
> >         .smp = smp_ops(bcm2836_smp_ops),
> > +       .init_early = bcm2835_init_early,
> 
> The sum of all these changes make me think, that we should start a new
> board for BCM2711 instead of extending BCM2835.
> 

I agree, I did it locally but merged it into the current board file to make the
patch smaller.

> Best regards
> Stefan Wahren
> 
> >  MACHINE_END
> > 
> >  Regards,
> >  Nicolas
> >
Stefan Wahren July 23, 2019, 9:30 p.m. UTC | #10
Hi Nicolas,

Am 23.07.19 um 19:33 schrieb Nicolas Saenz Julienne:
> On Tue, 2019-07-23 at 18:26 +0200, Stefan Wahren wrote:
>> Hi Nicolas,
>>
>> thanks for your work, but i'm a little bit sceptical about these
>> changes. So here some thoughts.
>>
>> Am 23.07.19 um 15:32 schrieb Nicolas Saenz Julienne:
>>> On Tue, 2019-07-23 at 11:34 +0200, Christoph Hellwig wrote:
>>>> On Mon, Jul 22, 2019 at 08:10:17PM +0200, Stefan Wahren wrote:
>>>>> i rebased this series also and got this only on the RPi 4.
>>>>>
>>>>> After reverting the following:
>>>>>
>>>>> 79a986721de dma-mapping: remove dma_max_pfn
>>>>> 7559d612dff0 mmc: core: let the dma map ops handle bouncing
>>>>>
>>>>> This crash disappear, but wifi seems to be still broken.
>>>>>
>>>>> Would be nice, if you can investigate further.
>>>> That means dma addressing on this system doesn't just work for some
>>>> memory, and the mmc bounce buffering was papering over that just for
>>>> mmc.  Do you have highmem on this system?
>>>>
>>>> You might want to try this series, which has been submitted upstream:
>>>>
>>>>
> http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/arm-swiotlb
>>> Hi Christoph,
>>> I tried your series on top of Stefan's, it has no effect. I guess it's no
>>> surprise as with mult_v7_defconfig, you get SWIOTLB=n & LPAE=n.
>>>
>>> FYI DMA addressing constraints for RPi4 are the following: devices can only
>>> access the first GB of ram even though the board might have up to 4GB of
>>> ram.
>>> The DMA addresses are aliased with a 0xc0000000 offset. So 0x00000000 phys
>>> is
>>> aliased to 0xc0000000 in DMA. This is the same as for an RFC you commented
>>> last
>>> week trying to fix similar issues for arm64.
>>>
>>> You state in "arm: use swiotlb for bounce buffer on LPAE configs" that "The
>>> DMA
>>> API requires that 32-bit DMA masks are always supported". If I understand it
>>> correctly this device breaks that assumption. Which implies we need a bounce
>>> buffer system in place for any straming DMA user.
>>>
>>> It seems we're unable to use dma-direct/swiotlb, so I enabled arm's
>>> dmabounce
>>> on all devices hooked into RPi's limited interconnect, which fixes this
>>> issue.
>> Does it fix the wifi issue too?
> Well it works as long as I revert this: 901bb98918 ("nl80211: require and
> validate vendor command policy"). Which has nothing to do with DMA anyways.
>
> Was this the issue you where seeing?
Yes. So it's a regression? I will try to test it with a RPi 3B+
Chen-Yu Tsai July 24, 2019, 2:53 a.m. UTC | #11
On Wed, Jul 24, 2019 at 1:33 AM Nicolas Saenz Julienne
<nsaenzjulienne@suse.de> wrote:
>
> On Tue, 2019-07-23 at 18:26 +0200, Stefan Wahren wrote:
> > Hi Nicolas,
> >
> > thanks for your work, but i'm a little bit sceptical about these
> > changes. So here some thoughts.
> >
> > Am 23.07.19 um 15:32 schrieb Nicolas Saenz Julienne:
> > > On Tue, 2019-07-23 at 11:34 +0200, Christoph Hellwig wrote:
> > > > On Mon, Jul 22, 2019 at 08:10:17PM +0200, Stefan Wahren wrote:
> > > > > i rebased this series also and got this only on the RPi 4.
> > > > >
> > > > > After reverting the following:
> > > > >
> > > > > 79a986721de dma-mapping: remove dma_max_pfn
> > > > > 7559d612dff0 mmc: core: let the dma map ops handle bouncing
> > > > >
> > > > > This crash disappear, but wifi seems to be still broken.
> > > > >
> > > > > Would be nice, if you can investigate further.
> > > > That means dma addressing on this system doesn't just work for some
> > > > memory, and the mmc bounce buffering was papering over that just for
> > > > mmc.  Do you have highmem on this system?
> > > >
> > > > You might want to try this series, which has been submitted upstream:
> > > >
> > > >
> http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/arm-swiotlb
> > > Hi Christoph,
> > > I tried your series on top of Stefan's, it has no effect. I guess it's no
> > > surprise as with mult_v7_defconfig, you get SWIOTLB=n & LPAE=n.
> > >
> > > FYI DMA addressing constraints for RPi4 are the following: devices can only
> > > access the first GB of ram even though the board might have up to 4GB of
> > > ram.
> > > The DMA addresses are aliased with a 0xc0000000 offset. So 0x00000000 phys
> > > is
> > > aliased to 0xc0000000 in DMA. This is the same as for an RFC you commented
> > > last
> > > week trying to fix similar issues for arm64.
> > >
> > > You state in "arm: use swiotlb for bounce buffer on LPAE configs" that "The
> > > DMA
> > > API requires that 32-bit DMA masks are always supported". If I understand it
> > > correctly this device breaks that assumption. Which implies we need a bounce
> > > buffer system in place for any straming DMA user.
> > >
> > > It seems we're unable to use dma-direct/swiotlb, so I enabled arm's
> > > dmabounce
> > > on all devices hooked into RPi's limited interconnect, which fixes this
> > > issue.
> > Does it fix the wifi issue too?
>
> Well it works as long as I revert this: 901bb98918 ("nl80211: require and
> validate vendor command policy"). Which has nothing to do with DMA anyways.
>
> Was this the issue you where seeing?
>
> [    4.969679] WARNING: CPU: 2 PID: 21 at net/wireless/core.c:868 wiphy_register+0x8e8/0xbdc [cfg80211]
> [...]
> [    4.969974] ieee80211 phy0: brcmf_cfg80211_attach: Could not register wiphy device (-22)

We're seeing this on different platforms (allwinner / rockchip / amlogic)
with Broadcom WiFi chips. So it's unlikely to be related with anything in
this series.

I believe a fix for this has already been queued up:

https://git.kernel.org/pub/scm/linux/kernel/git/jberg/mac80211.git/commit/?id=91046d6364afde646734c7ead1f649d253c386e9

ChenYu
Nicolas Saenz Julienne July 24, 2019, 8:37 a.m. UTC | #12
> > > Does it fix the wifi issue too?
> > 
> > Well it works as long as I revert this: 901bb98918 ("nl80211: require and
> > validate vendor command policy"). Which has nothing to do with DMA anyways.
> > 
> > Was this the issue you where seeing?
> > 
> > [    4.969679] WARNING: CPU: 2 PID: 21 at net/wireless/core.c:868
> > wiphy_register+0x8e8/0xbdc [cfg80211]
> > [...]
> > [    4.969974] ieee80211 phy0: brcmf_cfg80211_attach: Could not register
> > wiphy device (-22)
> 
> We're seeing this on different platforms (allwinner / rockchip / amlogic)
> with Broadcom WiFi chips. So it's unlikely to be related with anything in
> this series.
> 
> I believe a fix for this has already been queued up:
> 
> 
https://git.kernel.org/pub/scm/linux/kernel/git/jberg/mac80211.git/commit/?id=91046d6364afde646734c7ead1f649d253c386e9

Thanks for pointing it out, it fixes the issue alright.
Nicolas Saenz Julienne July 25, 2019, 8:22 a.m. UTC | #13
> > > Any thoughts on this?
> > > 
> > > diff --git a/arch/arm/mach-bcm/Kconfig b/arch/arm/mach-bcm/Kconfig
> > > index 5e5f1fabc3d4..3db8deed83a6 100644
> > > --- a/arch/arm/mach-bcm/Kconfig
> > > +++ b/arch/arm/mach-bcm/Kconfig
> > > @@ -168,6 +168,7 @@ config ARCH_BCM2835
> > >         select PINCTRL
> > >         select PINCTRL_BCM2835
> > >         select MFD_CORE
> > > +       select DMABOUNCE
> > >         help
> > >           This enables support for the Broadcom BCM2835 and BCM2836 SoCs.
> > >           This SoC is used in the Raspberry Pi and Roku 2 devices.
> > > diff --git a/arch/arm/mach-bcm/board_bcm2835.c b/arch/arm/mach-
> > > bcm/board_bcm2835.c
> > > index c09cf25596af..be788849c4bb 100644
> > > --- a/arch/arm/mach-bcm/board_bcm2835.c
> > > +++ b/arch/arm/mach-bcm/board_bcm2835.c
> > > @@ -3,6 +3,8 @@
> > >   * Copyright (C) 2010 Broadcom
> > >   */
> > > 
> > > +#include <linux/device.h>
> > > +#include <linux/dma-mapping.h>
> > >  #include <linux/init.h>
> > >  #include <linux/irqchip.h>
> > >  #include <linux/of_address.h>
> > > @@ -24,8 +26,37 @@ static const char * const bcm2835_compat[] = {
> > >         NULL
> > >  };
> > > 
> > > +static int bcm2835_needs_bounce(struct device *dev, dma_addr_t dma_addr,
> > > size_t size)
> > > +{
> > > +       /*
> > > +        * The accepted dma addresses are [0xc0000000, 0xffffffff] which
> > > map
> > > to
> > > +        * ram's [0x00000000, 0x3fffffff].
> > > +        */
> > > +       return dma_addr < 3ULL * SZ_1G;
> > > +}
> > > +
> > > +/*
> > > + * Setup DMA mask to 1GB on devices hanging from soc interconnect
> > > + */
> > > +static int bcm2835_platform_notify(struct device *dev)
> > > +{
> > > +       if (dev->parent && !strcmp("soc", dev_name(dev->parent))) {
> > > +               dev->dma_mask = &dev->coherent_dma_mask;
> > > +               dev->coherent_dma_mask = DMA_BIT_MASK(30); /* 1GB */
> > Shouldn't this come from the device tree?
> 
> Yes, actually I could use the 'dma-ranges' parsing code I suggested on the
> arm64 RFC. The same goes with 'dma_zone_size = SZ_1G', it ideally should be
> calculated based on the device-tree.
> 
> The way I see it I'm not sure it's worth the effort, in arm64 we have no
> choice
> as there are no board files. But here we seem to be the only ones with this
> specific DMA addressing constraint, so fixing it in arm/common doesn't seem
> like it's going to benefit anyone else. Let's see how the arm arch maintainers
> react though.
> 
> There is one catch though. I missed it earlier as I was excited to see the
> board boot, but some devices are failing to set their DMA masks:
> 
> [    1.989576] dwc2 fe980000.usb: can't set coherent DMA mask: -5
> 
> It seems that other users of dmabounce also implement their own
> dma_supported(). I have to look into it.

Sadly it seems there are some limitations in dmabounce I didn't take into
account earlier. Among other things it can't deal with HighMem out of the box
and even when trying to adapt it to our needs, fails to do so as it allocates
using GFP_ATOMIC, which rules out using the CMA when allocating coherent
memory. Sorry for the noise, I got carried away too soon.

I did a dirty hack hooking up dma-direct/swiotlb to the board. It seems to be
working fine after some tweaks in arm's dma_capable(). That said I want to test
it further before sending anything ;).
Stefan Wahren July 25, 2019, 10:09 p.m. UTC | #14
Am 25.07.19 um 10:22 schrieb Nicolas Saenz Julienne:
>>>> Any thoughts on this?
>>>>
>>>> diff --git a/arch/arm/mach-bcm/Kconfig b/arch/arm/mach-bcm/Kconfig
>>>> index 5e5f1fabc3d4..3db8deed83a6 100644
>>>> --- a/arch/arm/mach-bcm/Kconfig
>>>> +++ b/arch/arm/mach-bcm/Kconfig
>>>> @@ -168,6 +168,7 @@ config ARCH_BCM2835
>>>>         select PINCTRL
>>>>         select PINCTRL_BCM2835
>>>>         select MFD_CORE
>>>> +       select DMABOUNCE
>>>>         help
>>>>           This enables support for the Broadcom BCM2835 and BCM2836 SoCs.
>>>>           This SoC is used in the Raspberry Pi and Roku 2 devices.
>>>> diff --git a/arch/arm/mach-bcm/board_bcm2835.c b/arch/arm/mach-
>>>> bcm/board_bcm2835.c
>>>> index c09cf25596af..be788849c4bb 100644
>>>> --- a/arch/arm/mach-bcm/board_bcm2835.c
>>>> +++ b/arch/arm/mach-bcm/board_bcm2835.c
>>>> @@ -3,6 +3,8 @@
>>>>   * Copyright (C) 2010 Broadcom
>>>>   */
>>>>
>>>> +#include <linux/device.h>
>>>> +#include <linux/dma-mapping.h>
>>>>  #include <linux/init.h>
>>>>  #include <linux/irqchip.h>
>>>>  #include <linux/of_address.h>
>>>> @@ -24,8 +26,37 @@ static const char * const bcm2835_compat[] = {
>>>>         NULL
>>>>  };
>>>>
>>>> +static int bcm2835_needs_bounce(struct device *dev, dma_addr_t dma_addr,
>>>> size_t size)
>>>> +{
>>>> +       /*
>>>> +        * The accepted dma addresses are [0xc0000000, 0xffffffff] which
>>>> map
>>>> to
>>>> +        * ram's [0x00000000, 0x3fffffff].
>>>> +        */
>>>> +       return dma_addr < 3ULL * SZ_1G;
>>>> +}
>>>> +
>>>> +/*
>>>> + * Setup DMA mask to 1GB on devices hanging from soc interconnect
>>>> + */
>>>> +static int bcm2835_platform_notify(struct device *dev)
>>>> +{
>>>> +       if (dev->parent && !strcmp("soc", dev_name(dev->parent))) {
>>>> +               dev->dma_mask = &dev->coherent_dma_mask;
>>>> +               dev->coherent_dma_mask = DMA_BIT_MASK(30); /* 1GB */
>>> Shouldn't this come from the device tree?
>> Yes, actually I could use the 'dma-ranges' parsing code I suggested on the
>> arm64 RFC. The same goes with 'dma_zone_size = SZ_1G', it ideally should be
>> calculated based on the device-tree.
>>
>> The way I see it I'm not sure it's worth the effort, in arm64 we have no
>> choice
>> as there are no board files. But here we seem to be the only ones with this
>> specific DMA addressing constraint, so fixing it in arm/common doesn't seem
>> like it's going to benefit anyone else. Let's see how the arm arch maintainers
>> react though.
>>
>> There is one catch though. I missed it earlier as I was excited to see the
>> board boot, but some devices are failing to set their DMA masks:
>>
>> [    1.989576] dwc2 fe980000.usb: can't set coherent DMA mask: -5
>>
>> It seems that other users of dmabounce also implement their own
>> dma_supported(). I have to look into it.
> Sadly it seems there are some limitations in dmabounce I didn't take into
> account earlier. Among other things it can't deal with HighMem out of the box
> and even when trying to adapt it to our needs, fails to do so as it allocates
> using GFP_ATOMIC, which rules out using the CMA when allocating coherent
> memory. Sorry for the noise, I got carried away too soon.
>
> I did a dirty hack hooking up dma-direct/swiotlb to the board. It seems to be
> working fine after some tweaks in arm's dma_capable(). That said I want to test
> it further before sending anything ;).
>
Take your time.

But please inform about possible modifications to the devicetree. I will
sent out V2 of my series in a few days.
Stefan Wahren July 28, 2019, 11:22 a.m. UTC | #15
Hi Linus,
hi Michael,
hi Stephen,

Am 22.07.19 um 07:54 schrieb Stefan Wahren:
> This series adds minimal support for the new Raspberry Pi 4, so we are abl=
> e
> to login via debug UART.
>
> Patch 1-5:   Prepare platform and DTS for the new SoC BMC2711
> Patch 6-10:  Enable support for emmc2 on BCM2711
> Patch 11-12: Enable pinctrl for BCM2711
> Patch 13-17: Add Raspberry Pi 4 DTS support
> Patch 18:    Update MAINTAINERS
>
any comments on the clk / pinctrl parts or do you just wait for Rob's Ack?

Sorry to bother you, but i want to send a new version before my holidays.
Stefan Wahren July 29, 2019, 5:44 a.m. UTC | #16
Hi,

Am 27.07.19 um 17:17 schrieb Stefan Wahren:
> Hi,
>
> Am 24.07.19 um 10:37 schrieb Nicolas Saenz Julienne:
>>>>> Does it fix the wifi issue too?
>>>> Well it works as long as I revert this: 901bb98918 ("nl80211: require and
>>>> validate vendor command policy"). Which has nothing to do with DMA anyways.
>>>>
>>>> Was this the issue you where seeing?
>>>>
>>>> [    4.969679] WARNING: CPU: 2 PID: 21 at net/wireless/core.c:868
>>>> wiphy_register+0x8e8/0xbdc [cfg80211]
>>>> [...]
>>>> [    4.969974] ieee80211 phy0: brcmf_cfg80211_attach: Could not register
>>>> wiphy device (-22)
>>> We're seeing this on different platforms (allwinner / rockchip / amlogic)
>>> with Broadcom WiFi chips. So it's unlikely to be related with anything in
>>> this series.
>>>
>>> I believe a fix for this has already been queued up:
>>>
>>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/jberg/mac80211.git/commit/?id=91046d6364afde646734c7ead1f649d253c386e9
>>
>> Thanks for pointing it out, it fixes the issue alright.
>>
> i cannot confirm. I still need to revert Johannes' commit "nl80211:
> require and validate vendor command policy" to get brcmfmac probing on
> Raspberry Pi 3B+ and 4B.
>
> The commit "nl80211: fix VENDOR_CMD_RAW_DATA" didn't fix the probing
> issue (see warning above).
i rebased my series on top of Linux 5.3-rc2 and issue disappeared. I
assume the commit "wireless: fix nl80211 vendor commands" is also required.

Sorry, for the noise.
>
> Regards
> Stefan
>
>
Linus Walleij Aug. 5, 2019, 9:33 a.m. UTC | #17
On Sun, Jul 28, 2019 at 1:22 PM Stefan Wahren <wahrenst@gmx.net> wrote:

> any comments on the clk / pinctrl parts or do you just wait for Rob's Ack?

I'm just gonna apply it.

Yours,
Linus Walleij