| Message ID | 1569496834-7796-1-git-send-email-olekstysh@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | iommu/arm: Add Renesas IPMMU-VMSA support + Linux's iommu_fwspec | expand |
Hi, On 9/26/19 12:20 PM, Oleksandr Tyshchenko wrote: > Oleksandr Tyshchenko (8): > iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff > iommu/arm: Add ability to handle deferred probing request > xen/common: Introduce _xrealloc function > xen/common: Introduce xrealloc_flex_struct() helper macros > iommu/arm: Add lightweight iommu_fwspec support > iommu: Order the headers alphabetically in device_tree.c > iommu/arm: Introduce iommu_add_dt_device API > iommu/arm: Add Renesas IPMMU-VMSA support This series is now merged. Cheers,
On 26.09.19 17:56, Julien Grall wrote: > Hi, Hi Julien > > On 9/26/19 12:20 PM, Oleksandr Tyshchenko wrote: >> Oleksandr Tyshchenko (8): >> iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff >> iommu/arm: Add ability to handle deferred probing request >> xen/common: Introduce _xrealloc function >> xen/common: Introduce xrealloc_flex_struct() helper macros >> iommu/arm: Add lightweight iommu_fwspec support >> iommu: Order the headers alphabetically in device_tree.c >> iommu/arm: Introduce iommu_add_dt_device API >> iommu/arm: Add Renesas IPMMU-VMSA support > > This series is now merged. Thank you! Julien, what do you think regarding the following: https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg02304.html this one is intended to address "the main TODO" in the IPMMU-VMSA driver and it would be really nice if it could go in too... (I will be able to resolve any issues with it today/tomorrow if still present)
On Thu, 26 Sep 2019, Oleksandr wrote: > On 26.09.19 17:56, Julien Grall wrote: > > Hi, > > Hi Julien > > > > > > On 9/26/19 12:20 PM, Oleksandr Tyshchenko wrote: > > > Oleksandr Tyshchenko (8): > > > iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff > > > iommu/arm: Add ability to handle deferred probing request > > > xen/common: Introduce _xrealloc function > > > xen/common: Introduce xrealloc_flex_struct() helper macros > > > iommu/arm: Add lightweight iommu_fwspec support > > > iommu: Order the headers alphabetically in device_tree.c > > > iommu/arm: Introduce iommu_add_dt_device API > > > iommu/arm: Add Renesas IPMMU-VMSA support > > > > This series is now merged. > > Thank you! I just wanted to provide early feedback that this series causes problems with the legacy mmu-masters binding: (XEN) Failed to add /amba/dma@fd500000 to the IOMMU (XEN) Failed to add /amba/dma@fd510000 to the IOMMU (XEN) Failed to add /amba/dma@fd520000 to the IOMMU (XEN) Failed to add /amba/dma@fd530000 to the IOMMU (XEN) Failed to add /amba/dma@fd540000 to the IOMMU (XEN) Failed to add /amba/dma@fd550000 to the IOMMU (XEN) Failed to add /amba/dma@fd560000 to the IOMMU (XEN) Failed to add /amba/dma@fd570000 to the IOMMU (XEN) Failed to add /amba/dma@ffa80000 to the IOMMU (XEN) Failed to add /amba/dma@ffa90000 to the IOMMU (XEN) Failed to add /amba/dma@ffaa0000 to the IOMMU (XEN) Failed to add /amba/dma@ffab0000 to the IOMMU (XEN) Failed to add /amba/dma@ffac0000 to the IOMMU (XEN) Failed to add /amba/dma@ffad0000 to the IOMMU (XEN) Failed to add /amba/dma@ffae0000 to the IOMMU (XEN) Failed to add /amba/dma@ffaf0000 to the IOMMU (XEN) Failed to add /amba/nand@ff100000 to the IOMMU (XEN) Failed to add /amba/ethernet@ff0b0000 to the IOMMU (XEN) Failed to add /amba/ethernet@ff0c0000 to the IOMMU (XEN) Failed to add /amba/ethernet@ff0d0000 to the IOMMU (XEN) Failed to add /amba/spi@ff0f0000 to the IOMMU (XEN) Failed to add /amba/sdhci@ff160000 to the IOMMU (XEN) Failed to add /amba/sdhci@ff170000 to the IOMMU (XEN) Failed to add /amba/usb@fe200000 to the IOMMU (XEN) Failed to add /amba/usb@fe300000 to the IOMMU Booting fails and also even forcing to continue devices are unusable. I haven't managed to investigate further.
Hi Stefano Sorry for the possible format issues. сб, 28 сент. 2019 г., 01:50 Stefano Stabellini <sstabellini@kernel.org>: > On Thu, 26 Sep 2019, Oleksandr wrote: > > On 26.09.19 17:56, Julien Grall wrote: > > > Hi, > > > > Hi Julien > > > > > > > > > > On 9/26/19 12:20 PM, Oleksandr Tyshchenko wrote: > > > > Oleksandr Tyshchenko (8): > > > > iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs > stuff > > > > iommu/arm: Add ability to handle deferred probing request > > > > xen/common: Introduce _xrealloc function > > > > xen/common: Introduce xrealloc_flex_struct() helper macros > > > > iommu/arm: Add lightweight iommu_fwspec support > > > > iommu: Order the headers alphabetically in device_tree.c > > > > iommu/arm: Introduce iommu_add_dt_device API > > > > iommu/arm: Add Renesas IPMMU-VMSA support > > > > > > This series is now merged. > > > > Thank you! > > I just wanted to provide early feedback that this series causes problems > with the legacy mmu-masters binding: > This series was developed in a way to add new functionality, but not to brake existing (legacy bindings). Probably, I missed something important. iommu_add_dt_device() could return an error (I assume, this is what you are facing) if the device node in DT contains "iommus" property (I mean, uses new bindings), but the IOMMU driver doesn't implement required callbacks yet. Do the device nodes in your DT contain "iommus" property? And to which domain these devices (in your log) are going to be assigned (dom0 or other domains?). > (XEN) Failed to add /amba/dma@fd500000 to the IOMMU > (XEN) Failed to add /amba/dma@fd510000 to the IOMMU > (XEN) Failed to add /amba/dma@fd520000 to the IOMMU > (XEN) Failed to add /amba/dma@fd530000 to the IOMMU > (XEN) Failed to add /amba/dma@fd540000 to the IOMMU > (XEN) Failed to add /amba/dma@fd550000 to the IOMMU > (XEN) Failed to add /amba/dma@fd560000 to the IOMMU > (XEN) Failed to add /amba/dma@fd570000 to the IOMMU > (XEN) Failed to add /amba/dma@ffa80000 to the IOMMU > (XEN) Failed to add /amba/dma@ffa90000 to the IOMMU > (XEN) Failed to add /amba/dma@ffaa0000 to the IOMMU > (XEN) Failed to add /amba/dma@ffab0000 to the IOMMU > (XEN) Failed to add /amba/dma@ffac0000 to the IOMMU > (XEN) Failed to add /amba/dma@ffad0000 to the IOMMU > (XEN) Failed to add /amba/dma@ffae0000 to the IOMMU > (XEN) Failed to add /amba/dma@ffaf0000 to the IOMMU > (XEN) Failed to add /amba/nand@ff100000 to the IOMMU > (XEN) Failed to add /amba/ethernet@ff0b0000 to the IOMMU > (XEN) Failed to add /amba/ethernet@ff0c0000 to the IOMMU > (XEN) Failed to add /amba/ethernet@ff0d0000 to the IOMMU > (XEN) Failed to add /amba/spi@ff0f0000 to the IOMMU > (XEN) Failed to add /amba/sdhci@ff160000 to the IOMMU > (XEN) Failed to add /amba/sdhci@ff170000 to the IOMMU > (XEN) Failed to add /amba/usb@fe200000 to the IOMMU > (XEN) Failed to add /amba/usb@fe300000 to the IOMMU > > Booting fails and also even forcing to continue devices are unusable. I > haven't managed to investigate further.
On 28/09/2019 00:52, Oleksandr Tyshchenko wrote: > сб, 28 сент. 2019 г., 01:50 Stefano Stabellini <sstabellini@kernel.org > <mailto:sstabellini@kernel.org>>: > > On Thu, 26 Sep 2019, Oleksandr wrote: > > On 26.09.19 17:56, Julien Grall wrote: > > > Hi, > > > > Hi Julien > > > > > > > > > > On 9/26/19 12:20 PM, Oleksandr Tyshchenko wrote: > > > > Oleksandr Tyshchenko (8): > > > > iommu/arm: Add iommu_helpers.c file to keep common for > IOMMUs stuff > > > > iommu/arm: Add ability to handle deferred probing request > > > > xen/common: Introduce _xrealloc function > > > > xen/common: Introduce xrealloc_flex_struct() helper macros > > > > iommu/arm: Add lightweight iommu_fwspec support > > > > iommu: Order the headers alphabetically in device_tree.c > > > > iommu/arm: Introduce iommu_add_dt_device API > > > > iommu/arm: Add Renesas IPMMU-VMSA support > > > > > > This series is now merged. > > > > Thank you! > > I just wanted to provide early feedback that this series causes problems > with the legacy mmu-masters binding: > > > This series was developed in a way to add new functionality, but not to > brake existing (legacy bindings). Probably, I missed something > important. iommu_add_dt_device() could return an error (I assume, this > is what you are facing) if the device node in DT contains "iommus" > property (I mean, uses new bindings), but the IOMMU driver doesn't > implement required callbacks yet. Do the device nodes in your DT contain > "iommus" property? And to which domain these devices (in your log) are > going to be assigned (dom0 or other domains?). Looking at the bindings, I don't think it is legit to have a node using both legacy and generic binding together. If this is what happens, then I would consider it as unsupported. Cheers,
On Sat, 28 Sep 2019, Julien Grall wrote: > On 28/09/2019 00:52, Oleksandr Tyshchenko wrote: > > сб, 28 сент. 2019 г., 01:50 Stefano Stabellini <sstabellini@kernel.org > > <mailto:sstabellini@kernel.org>>: > > > > On Thu, 26 Sep 2019, Oleksandr wrote: > > > On 26.09.19 17:56, Julien Grall wrote: > > > > Hi, > > > > > > Hi Julien > > > > > > > > > > > > > > On 9/26/19 12:20 PM, Oleksandr Tyshchenko wrote: > > > > > Oleksandr Tyshchenko (8): > > > > > iommu/arm: Add iommu_helpers.c file to keep common for > > IOMMUs stuff > > > > > iommu/arm: Add ability to handle deferred probing request > > > > > xen/common: Introduce _xrealloc function > > > > > xen/common: Introduce xrealloc_flex_struct() helper macros > > > > > iommu/arm: Add lightweight iommu_fwspec support > > > > > iommu: Order the headers alphabetically in device_tree.c > > > > > iommu/arm: Introduce iommu_add_dt_device API > > > > > iommu/arm: Add Renesas IPMMU-VMSA support > > > > > > > > This series is now merged. > > > > > > Thank you! > > > > I just wanted to provide early feedback that this series causes problems > > with the legacy mmu-masters binding: > > > > > > This series was developed in a way to add new functionality, but not to > > brake existing (legacy bindings). Probably, I missed something > > important. iommu_add_dt_device() could return an error (I assume, this > > is what you are facing) if the device node in DT contains "iommus" > > property (I mean, uses new bindings), but the IOMMU driver doesn't > > implement required callbacks yet. Do the device nodes in your DT contain > > "iommus" property? And to which domain these devices (in your log) are > > going to be assigned (dom0 or other domains?). > > Looking at the bindings, I don't think it is legit to have a node using > both legacy and generic binding together. If this is what happens, then > I would consider it as unsupported. I have just sent a fix for this. The issue is that some Xilinx device trees expose both the legacy and generic bindings, however, only one set of bindings is supposed to be used, either the legacy or the generic bindings (not both!). Linux solves this problem by probing for the existence of "mmu-masters" (the legacy bindings property) very early on and disabling the generic bindings if "mmu-masters" is present. Something like that would work for Xen too, but I chatted with Julien and came up with something simpler. (Also keeping in mind that a new colleague of mine has just started working on generic bindings support for the ARM SMMU driver in Xen so this issue will go away soon). See: https://marc.info/?l=xen-devel&m=156987707614744
On 30.09.19 23:58, Stefano Stabellini wrote: Hi Stefano > On Sat, 28 Sep 2019, Julien Grall wrote: >> On 28/09/2019 00:52, Oleksandr Tyshchenko wrote: >>> сб, 28 сент. 2019 г., 01:50 Stefano Stabellini <sstabellini@kernel.org >>> <mailto:sstabellini@kernel.org>>: >>> >>> On Thu, 26 Sep 2019, Oleksandr wrote: >>> > On 26.09.19 17:56, Julien Grall wrote: >>> > > Hi, >>> > >>> > Hi Julien >>> > >>> > >>> > > >>> > > On 9/26/19 12:20 PM, Oleksandr Tyshchenko wrote: >>> > > > Oleksandr Tyshchenko (8): >>> > > > iommu/arm: Add iommu_helpers.c file to keep common for >>> IOMMUs stuff >>> > > > iommu/arm: Add ability to handle deferred probing request >>> > > > xen/common: Introduce _xrealloc function >>> > > > xen/common: Introduce xrealloc_flex_struct() helper macros >>> > > > iommu/arm: Add lightweight iommu_fwspec support >>> > > > iommu: Order the headers alphabetically in device_tree.c >>> > > > iommu/arm: Introduce iommu_add_dt_device API >>> > > > iommu/arm: Add Renesas IPMMU-VMSA support >>> > > >>> > > This series is now merged. >>> > >>> > Thank you! >>> >>> I just wanted to provide early feedback that this series causes problems >>> with the legacy mmu-masters binding: >>> >>> >>> This series was developed in a way to add new functionality, but not to >>> brake existing (legacy bindings). Probably, I missed something >>> important. iommu_add_dt_device() could return an error (I assume, this >>> is what you are facing) if the device node in DT contains "iommus" >>> property (I mean, uses new bindings), but the IOMMU driver doesn't >>> implement required callbacks yet. Do the device nodes in your DT contain >>> "iommus" property? And to which domain these devices (in your log) are >>> going to be assigned (dom0 or other domains?). >> Looking at the bindings, I don't think it is legit to have a node using >> both legacy and generic binding together. If this is what happens, then >> I would consider it as unsupported. > I have just sent a fix for this. > > The issue is that some Xilinx device trees expose both the legacy and > generic bindings, however, only one set of bindings is supposed to be > used, either the legacy or the generic bindings (not both!). I expected this could be a reason. > Linux > solves this problem by probing for the existence of "mmu-masters" (the > legacy bindings property) very early on and disabling the generic > bindings if "mmu-masters" is present. > > Something like that would work for Xen too, but I chatted with Julien > and came up with something simpler. (Also keeping in mind that a new > colleague of mine has just started working on generic bindings support > for the ARM SMMU driver in Xen so this issue will go away soon). See: > > https://marc.info/?l=xen-devel&m=156987707614744 Sounds good to me.
From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com> The purpose of this patch series is to add IPMMU-VMSA support to Xen on ARM. Besides new IOMMU driver, this series contains "iommu_fwspec" support and new API iommu_add_dt_device() for adding DT device to IOMMU and many other things. The IPMMU-VMSA is VMSA-compatible I/O Memory Management Unit (IOMMU) which provides address translation and access protection functionalities to processing units and interconnect networks. Please note, this driver is supposed to work only with newest R-Car Gen3 SoCs revisions which IPMMU hardware supports stage 2 translation table format and is able to use CPU's P2M table as is if one is 3-level page table (up to 40 bit IPA). ---------- This driver is based on Linux's IPMMU-VMSA driver from Renesas BSP: https://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git/tree/drivers/iommu/ipmmu-vmsa.c?h=v4.14.75-ltsi/rcar-3.9.6 and Xen's SMMU driver: xen/drivers/passthrough/arm/smmu.c Although Xen driver has a lot in common with Linux driver, it is not a "direct ported" copy and should be treated as such. The major differences compare to the Linux driver are: 1. Stage 1/Stage 2 translation. Linux driver supports Stage 1 translation only (with Stage 1 translation table format). It manages page table by itself. But Xen driver supports Stage 2 translation (with Stage 2 translation table format) to be able to share the P2M with the CPU. Stage 1 translation is always bypassed in Xen driver. So, Xen driver is supposed to be used with newest R-Car Gen3 SoC revisions only (H3 ES3.0, M3-W+, etc.) which IPMMU H/W supports stage 2 translation table format. 2. AArch64 support. Linux driver uses VMSAv8-32 mode, while Xen driver enables Armv8 VMSAv8-64 mode to cover up to 40 bit input address. 3. Context bank (sets of page table) usage. In Xen, each context bank is mapped to one Xen domain. So, all devices being pass throughed to the same Xen domain share the same context bank. 4. IPMMU device tracking. In Xen, all IPMMU devices are managed by single driver instance. So, driver uses global list to keep track of registered devices. ---------- Series was tested on R-Car Gen3 H3 ES3.0/M3N ES1.0 based boards using current staging (babde47 introduce a 'passthrough' configuration option to xl.cfg...) in a system with several DMA masters being assigned to different guest domains. Guest domain reboot, destroy/create are functional. You can find the whole series (and [1]) here: repo: https://github.com/otyshchenko1/xen.git branch: ipmmu_upstream6 You can find previous discussions here: [V1] https://lists.xenproject.org/archives/html/xen-devel/2019-06/msg01755.html [V2] https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg00253.html [V3] https://lists.xenproject.org/archives/html/xen-devel/2019-08/msg01948.html [V4] https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg01255.html [V5] https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg02282.html [1] Also, please note, there is a patch in ML which is intended to address the main TODO in the IPMMU-VMSA driver: https://lists.xenproject.org/archives/html/xen-devel/2019-09/msg02304.html For convenience: 1. iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff A-b 2. iommu/arm: Add ability to handle deferred probing request R-b 3. xen/common: Introduce _xrealloc function -none- 4. xen/common: Introduce xrealloc_flex_struct() helper macros R-b 5. iommu/arm: Add lightweight iommu_fwspec support A-b 6. iommu: Order the headers alphabetically in device_tree.c A-b 7. iommu/arm: Introduce iommu_add_dt_device API -none- 8. iommu/arm: Add Renesas IPMMU-VMSA support A-b/R-b Oleksandr Tyshchenko (8): iommu/arm: Add iommu_helpers.c file to keep common for IOMMUs stuff iommu/arm: Add ability to handle deferred probing request xen/common: Introduce _xrealloc function xen/common: Introduce xrealloc_flex_struct() helper macros iommu/arm: Add lightweight iommu_fwspec support iommu: Order the headers alphabetically in device_tree.c iommu/arm: Introduce iommu_add_dt_device API iommu/arm: Add Renesas IPMMU-VMSA support xen/arch/arm/domain_build.c | 25 +- xen/arch/arm/platforms/Kconfig | 1 + xen/common/xmalloc_tlsf.c | 114 ++- xen/drivers/passthrough/Kconfig | 13 + xen/drivers/passthrough/arm/Makefile | 3 +- xen/drivers/passthrough/arm/iommu.c | 56 +- xen/drivers/passthrough/arm/iommu_fwspec.c | 97 ++ xen/drivers/passthrough/arm/iommu_helpers.c | 84 ++ xen/drivers/passthrough/arm/ipmmu-vmsa.c | 1333 +++++++++++++++++++++++++++ xen/drivers/passthrough/arm/smmu.c | 48 +- xen/drivers/passthrough/device_tree.c | 83 +- xen/include/asm-arm/device.h | 7 +- xen/include/asm-arm/iommu.h | 10 + xen/include/asm-arm/iommu_fwspec.h | 68 ++ xen/include/xen/device_tree.h | 7 + xen/include/xen/iommu.h | 21 + xen/include/xen/xmalloc.h | 13 + 17 files changed, 1907 insertions(+), 76 deletions(-) create mode 100644 xen/drivers/passthrough/arm/iommu_fwspec.c create mode 100644 xen/drivers/passthrough/arm/iommu_helpers.c create mode 100644 xen/drivers/passthrough/arm/ipmmu-vmsa.c create mode 100644 xen/include/asm-arm/iommu_fwspec.h