mbox series

[00/14] iommu: Move iommu_fwspec out of 'struct device'

Message ID 20200228150820.15340-1-joro@8bytes.org (mailing list archive)
Headers show
Series iommu: Move iommu_fwspec out of 'struct device' | expand

Message

Joerg Roedel Feb. 28, 2020, 3:08 p.m. UTC 
Hi,

here is a patch-set to rename iommu_param to dev_iommu and
establish it as a struct for generic per-device iommu-data.
Also move the iommu_fwspec pointer from struct device into
dev_iommu to have less iommu-related pointers in struct
device.

The bigger part of this patch-set moves the iommu_priv
pointer from struct iommu_fwspec to dev_iommu, making is
usable for iommu-drivers which do not use fwspecs.

The changes for that were mostly straightforward, except for
the arm-smmu (_not_ arm-smmu-v3) and the qcom iommu driver.
Unfortunatly I don't have the hardware for those, so any
testing of these drivers is greatly appreciated.

I boot-tested these changes on arm-smmu-v3 enabled KVM VM.

Please review.

Regards,

	Joerg

Joerg Roedel (14):
  ACPI/IORT: Remove direct access of dev->iommu_fwspec
  drm/msm/mdp5: Remove direct access of dev->iommu_fwspec
  iommu/tegra-gart: Remove direct access of dev->iommu_fwspec
  iommu/arm-smmu: Fix uninitilized variable warning
  iommu: Rename struct iommu_param to dev_iommu
  iommu: Move iommu_fwspec to struct dev_iommu
  iommu: Introduce accessors for iommu private data
  iommu/arm-smmu-v3: Use accessor functions for iommu private data
  iommu/arm-smmu: Use accessor functions for iommu private data
  iommu/renesas: Use accessor functions for iommu private data
  iommu/mediatek: Use accessor functions for iommu private data
  iommu/qcom: Use accessor functions for iommu private data
  iommu/virtio: Use accessor functions for iommu private data
  iommu: Move fwspec->iommu_priv to struct dev_iommu

 drivers/acpi/arm64/iort.c                |  3 +-
 drivers/gpu/drm/msm/disp/mdp5/mdp5_kms.c |  2 +-
 drivers/iommu/arm-smmu-v3.c              | 10 ++--
 drivers/iommu/arm-smmu.c                 | 58 +++++++++++-----------
 drivers/iommu/iommu.c                    | 31 ++++++------
 drivers/iommu/ipmmu-vmsa.c               |  7 +--
 drivers/iommu/mtk_iommu.c                | 13 +++--
 drivers/iommu/mtk_iommu_v1.c             | 14 +++---
 drivers/iommu/qcom_iommu.c               | 61 ++++++++++++++----------
 drivers/iommu/tegra-gart.c               |  2 +-
 drivers/iommu/virtio-iommu.c             | 11 ++---
 include/linux/device.h                   |  7 ++-
 include/linux/iommu.h                    | 29 ++++++++---
 13 files changed, 137 insertions(+), 111 deletions(-)

Comments

Will Deacon March 3, 2020, 7:16 p.m. UTC | #1 
Hi Joerg,

On Fri, Feb 28, 2020 at 04:08:06PM +0100, Joerg Roedel wrote:
> here is a patch-set to rename iommu_param to dev_iommu and
> establish it as a struct for generic per-device iommu-data.
> Also move the iommu_fwspec pointer from struct device into
> dev_iommu to have less iommu-related pointers in struct
> device.
> 
> The bigger part of this patch-set moves the iommu_priv
> pointer from struct iommu_fwspec to dev_iommu, making is
> usable for iommu-drivers which do not use fwspecs.
> 
> The changes for that were mostly straightforward, except for
> the arm-smmu (_not_ arm-smmu-v3) and the qcom iommu driver.
> Unfortunatly I don't have the hardware for those, so any
> testing of these drivers is greatly appreciated.

I haven't had a chance to review this properly yet, but I did take it
for a spin on my Seattle board with MMU-400 (arm-smmu) and it seems to
work the same as before, so:

Tested-by: Will Deacon <will@kernel.org> # arm-smmu

I'll try to review the patches soon.

Cheers,

Will
Joerg Roedel March 4, 2020, 1:27 p.m. UTC | #2 
Hi Will,

On Tue, Mar 03, 2020 at 07:16:25PM +0000, Will Deacon wrote:
> I haven't had a chance to review this properly yet, but I did take it
> for a spin on my Seattle board with MMU-400 (arm-smmu) and it seems to
> work the same as before, so:
> 
> Tested-by: Will Deacon <will@kernel.org> # arm-smmu
> 
> I'll try to review the patches soon.

Thanks for testing it! I will send out a new version probably beginning
of next week (I am travelling this week) to fix the kbuild issue and
anything you might find.

Thanks,

	Joerg
Hanjun Guo March 6, 2020, 8:39 a.m. UTC | #3 
Hi Joerg,

On 2020/2/28 23:08, Joerg Roedel wrote:
> Hi,
> 
> here is a patch-set to rename iommu_param to dev_iommu and
> establish it as a struct for generic per-device iommu-data.
> Also move the iommu_fwspec pointer from struct device into
> dev_iommu to have less iommu-related pointers in struct
> device.
> 
> The bigger part of this patch-set moves the iommu_priv
> pointer from struct iommu_fwspec to dev_iommu, making is
> usable for iommu-drivers which do not use fwspecs.
> 
> The changes for that were mostly straightforward, except for
> the arm-smmu (_not_ arm-smmu-v3) and the qcom iommu driver.
> Unfortunatly I don't have the hardware for those, so any
> testing of these drivers is greatly appreciated.

I tested this patch set on Kunpeng 920 ARM64 server which
using smmu-v3 with ACPI booting, but triggered a NULL
pointer dereference and panic at boot:

[   14.832752] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000010
[   14.851425] Mem abort info:
[   14.858940]   ESR = 0x96000004
[   14.866519]   EC = 0x25: DABT (current EL), IL = 32 bits
[   14.876580]   SET = 0, FnV = 0
[   14.884412]   EA = 0, S1PTW = 0
[   14.892275] Data abort info:
[   14.899802]   ISV = 0, ISS = 0x00000004
[   14.908141]   CM = 0, WnR = 0
[   14.915401] [0000000000000010] user address but active_mm is swapper
[   14.926367] Internal error: Oops: 96000004 [#1] SMP
[   14.935992] Modules linked in:
[   14.943724] CPU: 36 PID: 1 Comm: swapper/0 Not tainted 5.6.0-rc2+ #4
[   14.955020] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 0.81 07/10/2019
[   14.972008] pstate: 80c00009 (Nzcv daif +PAN +UAO)
[   14.981774] pc : iort_iommu_configure+0xdc/0x230
[   14.991481] lr : iort_iommu_configure+0xcc/0x230
[   15.001075] sp : ffff800011b3bad0
[   15.009338] x29: ffff800011b3bad0 x28: ffff8000110a8968
[   15.019483] x27: ffff800011540000 x26: ffff8000110004c8
[   15.029390] x25: 0000000000000006 x24: 0000000000000000
[   15.039336] x23: 0000000000000000 x22: 0000000000000000
[   15.049270] x21: ffff8000113f9000 x20: ffff00002f5b0414
[   15.059038] x19: ffff002fdc8f90b0 x18: ffffffffffffffff
[   15.068590] x17: 0000000000000008 x16: 0000000000000005
[   15.078182] x15: ffff8000113f9948 x14: ffff2027d993e91c
[   15.087824] x13: ffff2027d993e16d x12: 0000000000000000
[   15.097440] x11: 0101010101010101 x10: 00000000ffffff76
[   15.106995] x9 : 0000000000000000 x8 : ffff2027d9a79b80
[   15.116629] x7 : 0000000000000000 x6 : 000000000000003f
[   15.126252] x5 : 0000000000000001 x4 : 0000000000000000
[   15.135781] x3 : 0000000000000600 x2 : 0000000000000040
[   15.145221] x1 : 0000000000000004 x0 : 0000000000000001
[   15.154472] Call trace:
[   15.160674]  iort_iommu_configure+0xdc/0x230
[   15.168752]  acpi_dma_configure+0x88/0xb8
[   15.176461]  pci_dma_configure+0xc0/0xe0
[   15.183935]  really_probe+0xbc/0x498
[   15.190853]  driver_probe_device+0x12c/0x148
[   15.198426]  device_driver_attach+0x74/0x98
[   15.205860]  __driver_attach+0xc4/0x178
[   15.213045]  bus_for_each_dev+0x84/0xd8
[   15.220185]  driver_attach+0x30/0x40
[   15.227051]  bus_add_driver+0x170/0x258
[   15.234241]  driver_register+0x64/0x118
[   15.241432]  __pci_register_driver+0x58/0x68
[   15.249202]  hibmc_pci_driver_init+0x28/0x30
[   15.256966]  do_one_initcall+0x54/0x250
[   15.264301]  kernel_init_freeable+0x24c/0x2e0
[   15.272164]  kernel_init+0x18/0x110
[   15.279068]  ret_from_fork+0x10/0x18
[   15.286033] Code: 2a0003f6 35000840 b9401a80 360005a0 (b94012e0)
[   15.295791] ---[ end trace 881fe61747538fd0 ]---
[   15.304039] Kernel panic - not syncing: Fatal exception

I don't have a time slot to do the detail investigation, but seems
that we don't have the iommu_fwspec properly initialized with ACPI
booting after applying this patch set.

Thanks
Hanjun
Jean-Philippe Brucker March 6, 2020, 10:09 a.m. UTC | #4 
On Fri, Mar 06, 2020 at 04:39:37PM +0800, Hanjun Guo wrote:
> Hi Joerg,
> 
> On 2020/2/28 23:08, Joerg Roedel wrote:
> > Hi,
> > 
> > here is a patch-set to rename iommu_param to dev_iommu and
> > establish it as a struct for generic per-device iommu-data.
> > Also move the iommu_fwspec pointer from struct device into
> > dev_iommu to have less iommu-related pointers in struct
> > device.
> > 
> > The bigger part of this patch-set moves the iommu_priv
> > pointer from struct iommu_fwspec to dev_iommu, making is
> > usable for iommu-drivers which do not use fwspecs.
> > 
> > The changes for that were mostly straightforward, except for
> > the arm-smmu (_not_ arm-smmu-v3) and the qcom iommu driver.
> > Unfortunatly I don't have the hardware for those, so any
> > testing of these drivers is greatly appreciated.
> 
> I tested this patch set on Kunpeng 920 ARM64 server which
> using smmu-v3 with ACPI booting, but triggered a NULL
> pointer dereference and panic at boot:

I think that's because patch 01/14 move the fwspec access too early. In 

                err = pci_for_each_dma_alias(to_pci_dev(dev),
                                             iort_pci_iommu_init, &info);

                if (!err && iort_pci_rc_supports_ats(node))
                        dev->iommu_fwspec->flags |= IOMMU_FWSPEC_PCI_RC_ATS;

the iommu_fwspec is only valid if iort_pci_iommu_init() initialized it
successfully, if err == 0. The following might fix it:

diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
index 0e981d7f3c7d..7d04424189df 100644
--- a/drivers/acpi/arm64/iort.c
+++ b/drivers/acpi/arm64/iort.c
@@ -1015,7 +1015,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 		return ops;

 	if (dev_is_pci(dev)) {
-		struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
+		struct iommu_fwspec *fwspec;
 		struct pci_bus *bus = to_pci_dev(dev)->bus;
 		struct iort_pci_alias_info info = { .dev = dev };

@@ -1028,7 +1028,8 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
 		err = pci_for_each_dma_alias(to_pci_dev(dev),
 					     iort_pci_iommu_init, &info);

-		if (!err && iort_pci_rc_supports_ats(node))
+		fwspec = dev_iommu_fwspec_get(dev);
+		if (fwspec && iort_pci_rc_supports_ats(node))
 			fwspec->flags |= IOMMU_FWSPEC_PCI_RC_ATS;
 	} else {
 		int i = 0;


Note that this use of iommu_fwspec will be removed by the ATS cleanup
series [1], but this change should work as a temporary fix.

Thanks,
Jean

[1] https://lore.kernel.org/linux-iommu/20200213165049.508908-10-jean-philippe@linaro.org/
Hanjun Guo March 6, 2020, 11:04 a.m. UTC | #5 
On 2020/3/6 18:09, Jean-Philippe Brucker wrote:
> On Fri, Mar 06, 2020 at 04:39:37PM +0800, Hanjun Guo wrote:
>> Hi Joerg,
>>
>> On 2020/2/28 23:08, Joerg Roedel wrote:
>>> Hi,
>>>
>>> here is a patch-set to rename iommu_param to dev_iommu and
>>> establish it as a struct for generic per-device iommu-data.
>>> Also move the iommu_fwspec pointer from struct device into
>>> dev_iommu to have less iommu-related pointers in struct
>>> device.
>>>
>>> The bigger part of this patch-set moves the iommu_priv
>>> pointer from struct iommu_fwspec to dev_iommu, making is
>>> usable for iommu-drivers which do not use fwspecs.
>>>
>>> The changes for that were mostly straightforward, except for
>>> the arm-smmu (_not_ arm-smmu-v3) and the qcom iommu driver.
>>> Unfortunatly I don't have the hardware for those, so any
>>> testing of these drivers is greatly appreciated.
>>
>> I tested this patch set on Kunpeng 920 ARM64 server which
>> using smmu-v3 with ACPI booting, but triggered a NULL
>> pointer dereference and panic at boot:
> 
> I think that's because patch 01/14 move the fwspec access too early. In 
> 
>                 err = pci_for_each_dma_alias(to_pci_dev(dev),
>                                              iort_pci_iommu_init, &info);
> 
>                 if (!err && iort_pci_rc_supports_ats(node))
>                         dev->iommu_fwspec->flags |= IOMMU_FWSPEC_PCI_RC_ATS;
> 
> the iommu_fwspec is only valid if iort_pci_iommu_init() initialized it
> successfully, if err == 0. The following might fix it:

Good catch :)

> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 0e981d7f3c7d..7d04424189df 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -1015,7 +1015,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  		return ops;
> 
>  	if (dev_is_pci(dev)) {
> -		struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
> +		struct iommu_fwspec *fwspec;
>  		struct pci_bus *bus = to_pci_dev(dev)->bus;
>  		struct iort_pci_alias_info info = { .dev = dev };
> 
> @@ -1028,7 +1028,8 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  		err = pci_for_each_dma_alias(to_pci_dev(dev),
>  					     iort_pci_iommu_init, &info);
> 
> -		if (!err && iort_pci_rc_supports_ats(node))
> +		fwspec = dev_iommu_fwspec_get(dev);
> +		if (fwspec && iort_pci_rc_supports_ats(node))
>  			fwspec->flags |= IOMMU_FWSPEC_PCI_RC_ATS;
>  	} else {
>  		int i = 0;

And the panic disappeared. Joerg, please feel free to add my Tested-by
for smmu-v3 and IORT ACPI patches with above changes.

> 
> 
> Note that this use of iommu_fwspec will be removed by the ATS cleanup
> series [1], but this change should work as a temporary fix.

Yes, as your patch set will set the ats_supported flag in the
host bridge level, not per device, nice cleanup.

Thanks
Hanjun
Joerg Roedel March 9, 2020, 4:32 p.m. UTC | #6 
Hi Jean-Philippe,

On Fri, Mar 06, 2020 at 11:09:55AM +0100, Jean-Philippe Brucker wrote:
> I think that's because patch 01/14 move the fwspec access too early. In 
> 
>                 err = pci_for_each_dma_alias(to_pci_dev(dev),
>                                              iort_pci_iommu_init, &info);
> 
>                 if (!err && iort_pci_rc_supports_ats(node))
>                         dev->iommu_fwspec->flags |= IOMMU_FWSPEC_PCI_RC_ATS;
> 
> the iommu_fwspec is only valid if iort_pci_iommu_init() initialized it
> successfully, if err == 0. The following might fix it:
> 
> diff --git a/drivers/acpi/arm64/iort.c b/drivers/acpi/arm64/iort.c
> index 0e981d7f3c7d..7d04424189df 100644
> --- a/drivers/acpi/arm64/iort.c
> +++ b/drivers/acpi/arm64/iort.c
> @@ -1015,7 +1015,7 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  		return ops;
> 
>  	if (dev_is_pci(dev)) {
> -		struct iommu_fwspec *fwspec = dev_iommu_fwspec_get(dev);
> +		struct iommu_fwspec *fwspec;
>  		struct pci_bus *bus = to_pci_dev(dev)->bus;
>  		struct iort_pci_alias_info info = { .dev = dev };
> 
> @@ -1028,7 +1028,8 @@ const struct iommu_ops *iort_iommu_configure(struct device *dev)
>  		err = pci_for_each_dma_alias(to_pci_dev(dev),
>  					     iort_pci_iommu_init, &info);
> 
> -		if (!err && iort_pci_rc_supports_ats(node))
> +		fwspec = dev_iommu_fwspec_get(dev);
> +		if (fwspec && iort_pci_rc_supports_ats(node))
>  			fwspec->flags |= IOMMU_FWSPEC_PCI_RC_ATS;
>  	} else {
>  		int i = 0;

Thanks a lot for the fix! I added it to patch 1/14.

Regards,

	Joerg