[v5,0/3] Apple M1 DART IOMMU driver

Message

Sven Peter Aug. 3, 2021, 12:16 p.m. UTC

Hi,

This is v5 of my Apple M1 DART IOMMU driver series as a follow up to the previous
versions [1][2][3][7].

Short summary: this series adds support for the iommu found in Apple's new M1
SoC which is required to use DMA on most peripherals like the display controller,
the USB ports or the internal PCIe bus (which is used for WiFi, Ethernet and
more USB ports).
So far this code has been tested by multiple people with dwc3 in host and
device mode (which both only require changes to the device tree after this
patchset) and PCIe (using a yet to be finalized patchset).

Note that this version has to be applied on top of iommu/core or iommu/next since
it now uses the new map_pages/unmap_pages API.


== Testing this patchset with the USB-C controller == 

The two USB-C ports on the M1 machines are exposed as two separate dwc3
controllers which are behind a DART. Now that my USB phy bringup code has been
merged into our bootloader m1n1 you can easily test this patchset yourself:

1) Follow the instructions at [4] to setup our bootloader m1n1 on your M1
   machine which will allow you to boot kernels using a normal USB cable.
   Note that you'll still need a special setup to expose the UART for very
   low-level debugging.

2) Apply this patchset and add the DART and dwc3 nodes as done in e.g. [5].

3) Boot the kernel through our bootloader m1n1. You'll need a version after
   commit [6] which enables the USB PHY and the USB PD chip.

Note that the dwc3 controller has a quirk where each root port can only be used
once right now. The most stable way to test is to already connected the USB
device(s) before booting the kernel.

It's also possible to test the PCIe bus but this requires a more complex setup
for now. I can write a quick howto if anyone is interested though.
(tl;dr: Mark Kettenis has a u-boot fork that includes PCIe bringup code and
Marc Zyngier has a WIP patchset to add a PCIe driver)

== Project Blurb ==

Asahi Linux is an open community project dedicated to developing and
maintaining mainline support for Apple Silicon on Linux. Feel free to
drop by #asahi and #asahi-dev on OFTC to chat with us, or check
our website for more information on the project:

https://asahilinux.org/

== Changes ==

Changes for v5:
 - Added reviewed-by and tested-by tags (thanks!)
 - Rebased on top of iommu/core and replaced map/unmap with map_pages/unmap_pages
 - Removed software bypass hacks: I've tried a few different variants now
   and they all have drawbacks or incompatibilities that I am not comfortable
   with. This means that PCIe devices (for which there is no kernel support yet
   anyway) will not work correctly for now on 4K kernels. I plan to address this
   in a follow-up series where I want to modify the dma-iommu layer to support
   pagesize mismatches.
 - Removed reference to ARM from the constants for the io-pgtable code
 - Addressed the following comments by Robin Murphy, which resulted in some major
   changes to apple-dart.c
   - Correctly assign iommu_groups in apple_dart_device_group
   - Get rid of the fwspec-inspired of_xlate linked lists and replaced them with
     a simple static array with a streamid bitmap covering all known cases
   - Relax locking: Now only a single spinlock around TLB flushes and a mutex
     around domain initialization are required. attach_dev/detach_dev uses
     atomic64_t.
   - Set .suppress_bind_attrs to prevent manual unbinding
   - Get rid of .shutdown since there's no real need to clean anything up
   - Manage interrupts manually instead of using devm_* to prevent situations
     where a shared interrupt could trigger while clocks are disabled
   - apple_dart_irq now prints "unknown" when more than a single error bit
     has been set.
   - Use DL_FLAG_AUTOREMOVE_SUPPLIER so that there's no need to keep track
     of the pointer
   - Ignore any unknown protection flags instead of failing
   - Use dev_err_ratelimited and clk_bulk_disable_unprepare instead of
     open-coding them
   - Correctly set and clear iommu_bus_ops
   - Removed unhelpful WARN_ONs and duplicate sanity checks
   - Removed unnecessary identity stream map reset code 
   - Renamed apple-dart-iommu.c to apple-dart.c
   - Fixed commit style to use the correct subsystem style
 - Possibly some smaller fixes I forgot about

Changes for v4:
 - Addressed Rob Herring's remark about the incorrect phandles in the device
   tree binding example and added his reviewed-by tag
 - Take the software linear mapping range from the bus instead of hardcoding
   it in the driver
 - Use def_domain_type to force bypass mode if there's a pagesize mismatch
   between the DART (hardwired to 16KB) and the kernel (may use 4K)
 - Added lockdep_assert_held instead of comments as suggested by Rouven Czerwinski
 - rebased on 5.13-rc7

Changes for v3:
 - fixed name of the iommu node in the device tree binding example
   pointed out by Arnd Bergmann
 - remove hardware specific checks from io-pgtable.c  as pointed out by
   Will Deacon
 - introduced a fake bypass mode by programming static linear pagetables
   if the DART does not support regular bypass mode as proposed by Alex
   Graf
 - added checks to enforce bypass mode if there is a pagesize mismatch
   between the DART HW and the CPU.
 - fixed usage of GFP_KERNEL during a held spinlock found by Julia Lawall
 - rebased on v5.13-rc3

Changes for v2:
 - fixed devicetree binding linting issues pointed out by Rob Herring and
   reworked that file.
 - made DART-specific code in io-pgtable.c unconditional and removed flag from
   Kconfig as proposed by Robin Murphy.
 - allowed multiple DART nodes in the "iommus" property as proposed by
   Rob Herring and Robin Murphy. this resulted in significant changes
   to apple-iommu-dart.c.
 - the domain aperture is now forced to 32bit if translation is enabled after
   the original suggestion to limit the aperture by Mark Kettenis and the
   follow-up discussion and investigation with Mark Kettenis, Arnd Bergmann,
   Robin Murphy and Rob Herring. This change also simplified the code
   in io-pgtable.c and made some of the improvements suggested during review
   not apply anymore.
 - added support for bypassed and isolated domain modes.
 - reject IOMMU_MMIO and IOMMU_NOEXEC since it's unknown how to set these up
   for now or if the hardware even supports these flags.
 - renamed some registers to be less confusing (mainly s/DOMAIN/STREAM/ to
   prevent confusion with linux's iommu domain concept).


[1] https://lore.kernel.org/linux-iommu/[e-mail]/
[2] https://lore.kernel.org/linux-iommu/[e-mail]/
[3] https://lore.kernel.org/linux-iommu/[e-mail]/
[4] https://github.com/AsahiLinux/docs/wiki/Developer-Quickstart
[5] https://github.com/AsahiLinux/linux/commit/7d4ebb0b22e9bfec849e2af86ddeb46ec29d7feb
[6] https://github.com/AsahiLinux/m1n1/commit/9529ec2b4fd6550f9cfd66d9f2448b90804699a1
[7] https://lore.kernel.org/linux-iommu/[e-mail]/

Sven Peter (3):
  iommu/io-pgtable: Add DART pagetable format
  dt-bindings: iommu: add DART iommu bindings
  iommu/dart: Add DART iommu driver

 .../devicetree/bindings/iommu/apple,dart.yaml |  81 ++
 MAINTAINERS                                   |   7 +
 drivers/iommu/Kconfig                         |  14 +
 drivers/iommu/Makefile                        |   1 +
 drivers/iommu/apple-dart.c                    | 923 ++++++++++++++++++
 drivers/iommu/io-pgtable-arm.c                |  63 ++
 drivers/iommu/io-pgtable.c                    |   1 +
 include/linux/io-pgtable.h                    |   7 +
 8 files changed, 1097 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/iommu/apple,dart.yaml
 create mode 100644 drivers/iommu/apple-dart.c

Comments

Joerg Roedel Aug. 9, 2021, 9:27 a.m. UTC | #1

On Tue, Aug 03, 2021 at 02:16:48PM +0200, Sven Peter wrote:
> Sven Peter (3):
>   iommu/io-pgtable: Add DART pagetable format
>   dt-bindings: iommu: add DART iommu bindings
>   iommu/dart: Add DART iommu driver

Applied, thanks. This driver now lives in the apple/dart branch of the
IOMMU tree.