| Message ID | 20231111184538.2371-1-brchuckz@aol.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | arm/mm: add option to prefer IOMMU ops for DMA on Xen | expand |
On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote: > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow > (and probably on other devices that use the Exynos mixer): > > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device > 14450000.mixer lacks support for IOMMU > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22 > exynos-drm exynos-drm: adev bind failed: -22 > exynos-dp: probe of 145b0000.dp-controller failed with error -22 > > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling > the new config option allows devices such as the Exynos mixer to use the > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error. > > The new config option is not set by default because it is likely some > devices that use IOMMU for DMA on Xen will cause DMA errors and memory > corruption when Xen PV block and network drivers are in use on the system. > > Link: https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/ > > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com> > --- > The reported error with the Exynos mixer is not fixed by default by adding > a second patch to select the new option in the Kconfig definition for the > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is > not certain setting the config option is suitable for all cases. So it is > necessary to explicitly select the new config option during the config > stage of the Linux kernel build to fix the reported error or similar > errors that have the same cause of lack of support for IOMMU on Xen. This > is necessary to avoid any regressions that might be caused by enabling the > new option by default for the Exynos mixer. > arch/arm/mm/dma-mapping.c | 6 ++++++ > drivers/xen/Kconfig | 16 ++++++++++++++++ > 2 files changed, 22 insertions(+) > > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c > index 5409225b4abc..ca04fdf01be3 100644 > --- a/arch/arm/mm/dma-mapping.c > +++ b/arch/arm/mm/dma-mapping.c > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, > if (iommu) > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent); > > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN FWIW I don't think this really needs a config option - if Xen *has* made an IOMMU available, then there isn't really much reason not to use it, and if for some reason someone really didn't want to then they could simply disable the IOMMU driver anyway. > + if (dev->dma_ops == &iommu_ops) { > + dev->archdata.dma_ops_setup = true; The existing assignment is effectively unconditional by this point anyway, so could probably just be moved earlier to save duplicating it (or perhaps just make the xen_setup_dma_ops() call conditional instead to save the early return as well). However, are the IOMMU DMA ops really compatible with Xen? The comments about hypercalls and foreign memory in xen_arch_need_swiotlb() leave me concerned that assuming non-coherent DMA to any old Dom0 page is OK might not actually work in general :/ Thanks, Robin. > + return; > + } > +#endif > xen_setup_dma_ops(dev); > dev->archdata.dma_ops_setup = true; > } > diff --git a/drivers/xen/Kconfig b/drivers/xen/Kconfig > index d5989871dd5d..44e1334b6acd 100644 > --- a/drivers/xen/Kconfig > +++ b/drivers/xen/Kconfig > @@ -181,6 +181,22 @@ config SWIOTLB_XEN > select DMA_OPS > select SWIOTLB > > +config ARM_DMA_USE_IOMMU_XEN > + bool "Prefer IOMMU DMA ops on Xen" > + depends on SWIOTLB_XEN > + depends on ARM_DMA_USE_IOMMU > + help > + Normally on Xen, the IOMMU is used by Xen and not exposed to > + Linux. Some Arm systems such as Exynos have an IOMMU that > + Xen does not use so the IOMMU is exposed to Linux in those > + cases. This option enables Linux to use the IOMMU instead of > + using the Xen swiotlb_dma_ops for DMA on Xen. > + > + Say N here unless support for one or more devices that use > + IOMMU ops instead of Xen swiotlb ops for DMA is needed and the > + devices that use the IOMMU do not cause any problems on the > + Xen system in use. > + > config XEN_PCI_STUB > bool >
On Tue, 14 Nov 2023, Robin Murphy wrote: > On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote: > > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when > > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow > > (and probably on other devices that use the Exynos mixer): > > > > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations > > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) > > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device > > 14450000.mixer lacks support for IOMMU > > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22 > > exynos-drm exynos-drm: adev bind failed: -22 > > exynos-dp: probe of 145b0000.dp-controller failed with error -22 > > > > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when > > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling > > the new config option allows devices such as the Exynos mixer to use the > > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error. > > > > The new config option is not set by default because it is likely some > > devices that use IOMMU for DMA on Xen will cause DMA errors and memory > > corruption when Xen PV block and network drivers are in use on the system. > > > > Link: > > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/ > > > > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com> > > --- > > The reported error with the Exynos mixer is not fixed by default by adding > > a second patch to select the new option in the Kconfig definition for the > > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is > > not certain setting the config option is suitable for all cases. So it is > > necessary to explicitly select the new config option during the config > > stage of the Linux kernel build to fix the reported error or similar > > errors that have the same cause of lack of support for IOMMU on Xen. This > > is necessary to avoid any regressions that might be caused by enabling the > > new option by default for the Exynos mixer. > > arch/arm/mm/dma-mapping.c | 6 ++++++ > > drivers/xen/Kconfig | 16 ++++++++++++++++ > > 2 files changed, 22 insertions(+) > > > > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c > > index 5409225b4abc..ca04fdf01be3 100644 > > --- a/arch/arm/mm/dma-mapping.c > > +++ b/arch/arm/mm/dma-mapping.c > > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64 > > dma_base, u64 size, > > if (iommu) > > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent); > > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN > > FWIW I don't think this really needs a config option - if Xen *has* made an > IOMMU available, then there isn't really much reason not to use it, and if for > some reason someone really didn't want to then they could simply disable the > IOMMU driver anyway. The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on purpose, it happens by accident. Certain things are going to break, specifically I am fairly certain PV drivers are going to break. If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but doesn't have a driver for.) I think it is OK for Chuck and others to play around with this configuration but I wouldn't add a new kconfig option to Linux to support it. If we do want a kconfig option, I would add a kconfig option or Linux command line option to enable/disable swiotlb-xen. Basically a way to force-enable or force-disable xen_swiotlb_detect(). That could be generally useful for debugging and would also solve the problem here as it could be used to force-disable swiotlb-xen. I would imagine that the end result is the same: the default ops (iommu_ops) are used. > > + if (dev->dma_ops == &iommu_ops) { > > + dev->archdata.dma_ops_setup = true; > > The existing assignment is effectively unconditional by this point anyway, so > could probably just be moved earlier to save duplicating it (or perhaps just > make the xen_setup_dma_ops() call conditional instead to save the early return > as well). > > However, are the IOMMU DMA ops really compatible with Xen? The comments about > hypercalls and foreign memory in xen_arch_need_swiotlb() leave me concerned > that assuming non-coherent DMA to any old Dom0 page is OK might not actually > work in general :/ Xen has (not yet upstreaming) support for nested IOMMU (Xen uses the IOMMU while also it exposes a virtual IOMMU to guests.) In those cases the iommu_ops should be compatible with Xen. swiotlb-xen is useful in cases where there is no IOMMU on the platform (or the IOMMU doesn't cover all DMA-capable devices) and Dom0 is 1:1 mapped. See include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect. If Dom0 is not 1:1 mapped swiotlb-xen doesn't work. If an IOMMU is present and covers all DMA-capable devices, then swiotlb-xen is superfluous. This last case is the interesting case for virtual IOMMU and Linux usage of iommu_ops.
On 11/14/2023 5:20 PM, Stefano Stabellini wrote: > On Tue, 14 Nov 2023, Robin Murphy wrote: >> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote: >> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when >> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow >> > (and probably on other devices that use the Exynos mixer): >> > >> > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations >> > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) >> > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device >> > 14450000.mixer lacks support for IOMMU >> > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22 >> > exynos-drm exynos-drm: adev bind failed: -22 >> > exynos-dp: probe of 145b0000.dp-controller failed with error -22 >> > >> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when >> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling >> > the new config option allows devices such as the Exynos mixer to use the >> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error. >> > >> > The new config option is not set by default because it is likely some >> > devices that use IOMMU for DMA on Xen will cause DMA errors and memory >> > corruption when Xen PV block and network drivers are in use on the system. >> > >> > Link: >> > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/ >> > >> > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com> >> > --- >> > The reported error with the Exynos mixer is not fixed by default by adding >> > a second patch to select the new option in the Kconfig definition for the >> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is >> > not certain setting the config option is suitable for all cases. So it is >> > necessary to explicitly select the new config option during the config >> > stage of the Linux kernel build to fix the reported error or similar >> > errors that have the same cause of lack of support for IOMMU on Xen. This >> > is necessary to avoid any regressions that might be caused by enabling the >> > new option by default for the Exynos mixer. >> > arch/arm/mm/dma-mapping.c | 6 ++++++ >> > drivers/xen/Kconfig | 16 ++++++++++++++++ >> > 2 files changed, 22 insertions(+) >> > >> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c >> > index 5409225b4abc..ca04fdf01be3 100644 >> > --- a/arch/arm/mm/dma-mapping.c >> > +++ b/arch/arm/mm/dma-mapping.c >> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64 >> > dma_base, u64 size, >> > if (iommu) >> > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent); >> > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN >> >> FWIW I don't think this really needs a config option - if Xen *has* made an >> IOMMU available, then there isn't really much reason not to use it, and if for >> some reason someone really didn't want to then they could simply disable the >> IOMMU driver anyway. > > The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen > doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has > a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on > purpose, it happens by accident. Certain things are going to break, > specifically I am fairly certain PV drivers are going to break. > > If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it > from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but > doesn't have a driver for.) > > I think it is OK for Chuck and others to play around with this > configuration but I wouldn't add a new kconfig option to Linux to > support it. > > If we do want a kconfig option, I would add a kconfig option or Linux > command line option to enable/disable swiotlb-xen. Basically a way to > force-enable or force-disable xen_swiotlb_detect(). That could be > generally useful for debugging and would also solve the problem here as > it could be used to force-disable swiotlb-xen. I would imagine that the > end result is the same: the default ops (iommu_ops) are used. I will try this. It isn't exactly what I have tested until now because in all my tests so far all the DMA capable devices on the Chromebook use swioltlb-xen except for the two devices that need to use the Exynos IOMMU to fix the error with the Exynos mixer. > > > >> > + if (dev->dma_ops == &iommu_ops) { >> > + dev->archdata.dma_ops_setup = true; >> >> The existing assignment is effectively unconditional by this point anyway, so >> could probably just be moved earlier to save duplicating it (or perhaps just >> make the xen_setup_dma_ops() call conditional instead to save the early return >> as well). >> >> However, are the IOMMU DMA ops really compatible with Xen? The comments about >> hypercalls and foreign memory in xen_arch_need_swiotlb() leave me concerned >> that assuming non-coherent DMA to any old Dom0 page is OK might not actually >> work in general :/ > > Xen has (not yet upstreaming) support for nested IOMMU (Xen uses the > IOMMU while also it exposes a virtual IOMMU to guests.) In those cases > the iommu_ops should be compatible with Xen. > > swiotlb-xen is useful in cases where there is no IOMMU on the platform > (or the IOMMU doesn't cover all DMA-capable devices) and Dom0 is 1:1 > mapped. See include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect. If Dom0 is > not 1:1 mapped swiotlb-xen doesn't work. If an IOMMU is present and > covers all DMA-capable devices, then swiotlb-xen is superfluous. It seems that swiotlb-xen works on this Chromebook since all but two of the DMA capable devices use it when configured with the Kconfig option added here and it seems to work fine so I presume Dom0 is 1:1 mapped as expected. It is possible that on this device, the IOMMU is only covering the two devices that need to use the Exynos IOMMU in the tests I have done. There are many other DMA capable devices that use swiotlb-xen DMA ops on Xen, but I have not checked what DMA ops the other devices use when Linux runs on the Chromebook on bare metal without Xen. So I plan to do some tests and see what DMA ops the other devices use if swiotlb-xen is disabled and also what DMA ops the other devices use when Linux runs on the Chromebook on bare metal without Xen. If these tests show the problem can be fixed by disabling swiotlb-xen with a Kconfig or command line option, I will propose v2 to implement that as a solution. > This last case is the interesting case for virtual IOMMU and Linux usage of > iommu_ops.
---> So I plan to do some tests and see what DMA ops the other devices use if swiotlb-xen is disabled and also what DMA ops the other devices use when Linux runs on the Chromebook on bare metal without Xen. If these tests show the problem can be fixed by disabling swiotlb-xen with a Kconfig or command line option, I will propose v2 to implement that as a solution. and this could bring you to the next level of our project. Try to install xen on different devices. At least it is my next project. I've already bought two arm64 phones where xen can be installed because there is a hack to overcome the bootloader / hypervisor protection mechanism. For sure I hope that you also want to buy them to work on this together. And don't worry about how much money they will cost you. I've bought them used and refurbished. Or you could buy only one,that I suggest could be the SM-A600G (Samsung Galaxy A6) with Exynos7870. Please start looking for it at a good price. On Wed, Nov 15, 2023 at 6:57 PM Chuck Zmudzinski <brchuckz@netscape.net> wrote: > On 11/14/2023 5:20 PM, Stefano Stabellini wrote: > > On Tue, 14 Nov 2023, Robin Murphy wrote: > >> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote: > >> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when > >> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow > >> > (and probably on other devices that use the Exynos mixer): > >> > > >> > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping > operations > >> > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) > >> > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* > Device > >> > 14450000.mixer lacks support for IOMMU > >> > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops > 0xc0d97554): -22 > >> > exynos-drm exynos-drm: adev bind failed: -22 > >> > exynos-dp: probe of 145b0000.dp-controller failed with error -22 > >> > > >> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when > >> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. > Enabling > >> > the new config option allows devices such as the Exynos mixer to use > the > >> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error. > >> > > >> > The new config option is not set by default because it is likely some > >> > devices that use IOMMU for DMA on Xen will cause DMA errors and memory > >> > corruption when Xen PV block and network drivers are in use on the > system. > >> > > >> > Link: > >> > > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/ > >> > > >> > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com> > >> > --- > >> > The reported error with the Exynos mixer is not fixed by default by > adding > >> > a second patch to select the new option in the Kconfig definition for > the > >> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is > >> > not certain setting the config option is suitable for all cases. So > it is > >> > necessary to explicitly select the new config option during the config > >> > stage of the Linux kernel build to fix the reported error or similar > >> > errors that have the same cause of lack of support for IOMMU on Xen. > This > >> > is necessary to avoid any regressions that might be caused by > enabling the > >> > new option by default for the Exynos mixer. > >> > arch/arm/mm/dma-mapping.c | 6 ++++++ > >> > drivers/xen/Kconfig | 16 ++++++++++++++++ > >> > 2 files changed, 22 insertions(+) > >> > > >> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c > >> > index 5409225b4abc..ca04fdf01be3 100644 > >> > --- a/arch/arm/mm/dma-mapping.c > >> > +++ b/arch/arm/mm/dma-mapping.c > >> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64 > >> > dma_base, u64 size, > >> > if (iommu) > >> > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, > coherent); > >> > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN > >> > >> FWIW I don't think this really needs a config option - if Xen *has* > made an > >> IOMMU available, then there isn't really much reason not to use it, and > if for > >> some reason someone really didn't want to then they could simply > disable the > >> IOMMU driver anyway. > > > > The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen > > doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has > > a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on > > purpose, it happens by accident. Certain things are going to break, > > specifically I am fairly certain PV drivers are going to break. > > > > If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it > > from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but > > doesn't have a driver for.) > > > > I think it is OK for Chuck and others to play around with this > > configuration but I wouldn't add a new kconfig option to Linux to > > support it. > > > > If we do want a kconfig option, I would add a kconfig option or Linux > > command line option to enable/disable swiotlb-xen. Basically a way to > > force-enable or force-disable xen_swiotlb_detect(). That could be > > generally useful for debugging and would also solve the problem here as > > it could be used to force-disable swiotlb-xen. I would imagine that the > > end result is the same: the default ops (iommu_ops) are used. > > I will try this. It isn't exactly what I have tested until now because > in all my tests so far all the DMA capable devices on the Chromebook use > swioltlb-xen except for the two devices that need to use the Exynos IOMMU > to fix the error with the Exynos mixer. > > > > > > > > >> > + if (dev->dma_ops == &iommu_ops) { > >> > + dev->archdata.dma_ops_setup = true; > >> > >> The existing assignment is effectively unconditional by this point > anyway, so > >> could probably just be moved earlier to save duplicating it (or perhaps > just > >> make the xen_setup_dma_ops() call conditional instead to save the early > return > >> as well). > >> > >> However, are the IOMMU DMA ops really compatible with Xen? The comments > about > >> hypercalls and foreign memory in xen_arch_need_swiotlb() leave me > concerned > >> that assuming non-coherent DMA to any old Dom0 page is OK might not > actually > >> work in general :/ > > > > Xen has (not yet upstreaming) support for nested IOMMU (Xen uses the > > IOMMU while also it exposes a virtual IOMMU to guests.) In those cases > > the iommu_ops should be compatible with Xen. > > > > swiotlb-xen is useful in cases where there is no IOMMU on the platform > > (or the IOMMU doesn't cover all DMA-capable devices) and Dom0 is 1:1 > > mapped. See include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect. If Dom0 is > > not 1:1 mapped swiotlb-xen doesn't work. If an IOMMU is present and > > covers all DMA-capable devices, then swiotlb-xen is superfluous. > > It seems that swiotlb-xen works on this Chromebook since all but two > of the DMA capable devices use it when configured with the Kconfig option > added here and it seems to work fine so I presume Dom0 is 1:1 mapped as > expected. It is possible that on this device, the IOMMU is only covering > the two devices that need to use the Exynos IOMMU in the tests I have done. > There are many other DMA capable devices that use swiotlb-xen DMA ops > on Xen, but I have not checked what DMA ops the other devices use when > Linux runs on the Chromebook on bare metal without Xen. > > So I plan to do some tests and see what DMA ops the other devices use if > swiotlb-xen is disabled and also what DMA ops the other devices use when > Linux runs on the Chromebook on bare metal without Xen. If these tests > show the problem can be fixed by disabling swiotlb-xen with a Kconfig or > command line option, I will propose v2 to implement that as a solution. > > > This last case is the interesting case for virtual IOMMU and Linux usage > of > > iommu_ops. >
Hello to everyone.
I'm trying to boot Linux 6.1.y as Xen dom0 on the Chromebook xe303c12, aka
Snow and configure and start a very basic domU guest,following the Chuck's
tutorial,located here :
https://github.com/mobile-virt/u-boot-chromebook-xe303c12/tree/chromebook/xen#starting-a-domu-guest
What I did has been to carefully follow his instructions,but I haven't
found a solution to fix this problem,yet :
# sudo xl create devuan.cfg -c
Parsing config from devuan.cfg
libxl: error: libxl_create.c:720:libxl__domain_make: domain creation fail:
Permission denied
libxl: error: libxl_create.c:1309:initiate_domain_create: cannot make
domain: -3
This is my devuan.cfg file :
kernel = '/Dati/xen/kernels/zImage-6.1.59-stb-xen-cbe+'
memory = '512'
name = 'Devuan'
vcpus = '1'
disk = [ '/Dati/xen/devuan.img,,xvda,w' ]
extra = 'console=hvc0 root=/dev/xvda rw init=/sbin/init
xen-fbfront.video=24,1024,768'
(I have tried also with root=/dev/xvda1 and root=/dev/xvda2,but leaving
disk = [ '/Dati/xen/devuan.img,,xvda,w' ] and not xvda1 or 2)
I have no idea about the reason(s) I always get that error,but I don't
think it is caused by a wrong creation of the devuan.img file. Can someone
point me in the right direction to understand what could be wrong ? I
haven't found any useful information on the internet.
This is bootxen.scr file where I have configured dom0_mem=768 :
mmc dev 1
ext2load mmc 1:3 0x42000000 zImage-6.6.0-xen-iommu-dma-on-xen
ext2load mmc 1:3 0x51000000 xen-4.17-armhf-armmp-0x51004000.ub
ext2load mmc 1:3 0x5ffec000 exynos5250-snow.dtb
fdt addr 0x5ffec000
fdt resize 1024
fdt set /chosen \#address-cells <0x2>
fdt set /chosen \#size-cells <0x2>
fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0
dom0_mem=768M dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=null"
fdt mknod /chosen dom0
fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module"
"multiboot,module"
fdt set /chosen/dom0 reg <0x0 0x42000000 0x0 0x87C200 >
fdt set /chosen xen,dom0-bootargs "console=tty1 root=/dev/mmcblk1p4 rw
rootwait clk_ignore_unused --no-log"
bootm 0x51000000 - 0x5ffec000
and I've rebooted the Chromebook using this command :
SMDK5250 # mmc dev 1
SMDK5250 # ext2load mmc 1:3 0x50000000 bootxen.scr; source 0x50000000
This is the memory available on the machine after having booted the
machine ready for xen :
# free -m
total used free shared buff/cache
available
Mem: 741 329 108 7 332
412
Swap: 0 0 0
Thanks in advance for any support.
On Wed, Nov 15, 2023 at 8:41 PM Mario Marietto <marietto2008@gmail.com>
wrote:
> ---> So I plan to do some tests and see what DMA ops the other devices use
> if swiotlb-xen is disabled and also what DMA ops the other devices use when
> Linux runs on the Chromebook on bare metal without Xen. If these tests show
> the problem can be fixed by disabling swiotlb-xen with a Kconfig or
> command line option, I will propose v2 to implement that as a solution.
>
> and this could bring you to the next level of our project. Try to install
> xen on different devices. At least it is my next project. I've already
> bought two arm64 phones where xen can be installed because there is a hack
> to overcome the bootloader / hypervisor protection mechanism. For sure I
> hope that you also want to buy them to work on this together. And don't
> worry about how much money they will cost you. I've bought them used and
> refurbished. Or you could buy only one,that I suggest could be the SM-A600G
> (Samsung Galaxy A6) with Exynos7870. Please start looking for it at a good
> price.
>
> On Wed, Nov 15, 2023 at 6:57 PM Chuck Zmudzinski <brchuckz@netscape.net>
> wrote:
>
>> On 11/14/2023 5:20 PM, Stefano Stabellini wrote:
>> > On Tue, 14 Nov 2023, Robin Murphy wrote:
>> >> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote:
>> >> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when
>> >> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook
>> Snow
>> >> > (and probably on other devices that use the Exynos mixer):
>> >> >
>> >> > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping
>> operations
>> >> > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354)
>> >> > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR*
>> Device
>> >> > 14450000.mixer lacks support for IOMMU
>> >> > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops
>> 0xc0d97554): -22
>> >> > exynos-drm exynos-drm: adev bind failed: -22
>> >> > exynos-dp: probe of 145b0000.dp-controller failed with error -22
>> >> >
>> >> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when
>> >> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux.
>> Enabling
>> >> > the new config option allows devices such as the Exynos mixer to use
>> the
>> >> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the
>> error.
>> >> >
>> >> > The new config option is not set by default because it is likely some
>> >> > devices that use IOMMU for DMA on Xen will cause DMA errors and
>> memory
>> >> > corruption when Xen PV block and network drivers are in use on the
>> system.
>> >> >
>> >> > Link:
>> >> >
>> https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/
>> >> >
>> >> > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com>
>> >> > ---
>> >> > The reported error with the Exynos mixer is not fixed by default by
>> adding
>> >> > a second patch to select the new option in the Kconfig definition
>> for the
>> >> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it
>> is
>> >> > not certain setting the config option is suitable for all cases. So
>> it is
>> >> > necessary to explicitly select the new config option during the
>> config
>> >> > stage of the Linux kernel build to fix the reported error or similar
>> >> > errors that have the same cause of lack of support for IOMMU on Xen.
>> This
>> >> > is necessary to avoid any regressions that might be caused by
>> enabling the
>> >> > new option by default for the Exynos mixer.
>> >> > arch/arm/mm/dma-mapping.c | 6 ++++++
>> >> > drivers/xen/Kconfig | 16 ++++++++++++++++
>> >> > 2 files changed, 22 insertions(+)
>> >> >
>> >> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c
>> >> > index 5409225b4abc..ca04fdf01be3 100644
>> >> > --- a/arch/arm/mm/dma-mapping.c
>> >> > +++ b/arch/arm/mm/dma-mapping.c
>> >> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev,
>> u64
>> >> > dma_base, u64 size,
>> >> > if (iommu)
>> >> > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu,
>> coherent);
>> >> > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN
>> >>
>> >> FWIW I don't think this really needs a config option - if Xen *has*
>> made an
>> >> IOMMU available, then there isn't really much reason not to use it,
>> and if for
>> >> some reason someone really didn't want to then they could simply
>> disable the
>> >> IOMMU driver anyway.
>> >
>> > The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen
>> > doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has
>> > a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on
>> > purpose, it happens by accident. Certain things are going to break,
>> > specifically I am fairly certain PV drivers are going to break.
>> >
>> > If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it
>> > from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but
>> > doesn't have a driver for.)
>> >
>> > I think it is OK for Chuck and others to play around with this
>> > configuration but I wouldn't add a new kconfig option to Linux to
>> > support it.
>> >
>> > If we do want a kconfig option, I would add a kconfig option or Linux
>> > command line option to enable/disable swiotlb-xen. Basically a way to
>> > force-enable or force-disable xen_swiotlb_detect(). That could be
>> > generally useful for debugging and would also solve the problem here as
>> > it could be used to force-disable swiotlb-xen. I would imagine that the
>> > end result is the same: the default ops (iommu_ops) are used.
>>
>> I will try this. It isn't exactly what I have tested until now because
>> in all my tests so far all the DMA capable devices on the Chromebook use
>> swioltlb-xen except for the two devices that need to use the Exynos IOMMU
>> to fix the error with the Exynos mixer.
>>
>> >
>> >
>> >
>> >> > + if (dev->dma_ops == &iommu_ops) {
>> >> > + dev->archdata.dma_ops_setup = true;
>> >>
>> >> The existing assignment is effectively unconditional by this point
>> anyway, so
>> >> could probably just be moved earlier to save duplicating it (or
>> perhaps just
>> >> make the xen_setup_dma_ops() call conditional instead to save the
>> early return
>> >> as well).
>> >>
>> >> However, are the IOMMU DMA ops really compatible with Xen? The
>> comments about
>> >> hypercalls and foreign memory in xen_arch_need_swiotlb() leave me
>> concerned
>> >> that assuming non-coherent DMA to any old Dom0 page is OK might not
>> actually
>> >> work in general :/
>> >
>> > Xen has (not yet upstreaming) support for nested IOMMU (Xen uses the
>> > IOMMU while also it exposes a virtual IOMMU to guests.) In those cases
>> > the iommu_ops should be compatible with Xen.
>> >
>> > swiotlb-xen is useful in cases where there is no IOMMU on the platform
>> > (or the IOMMU doesn't cover all DMA-capable devices) and Dom0 is 1:1
>> > mapped. See include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect. If Dom0 is
>> > not 1:1 mapped swiotlb-xen doesn't work. If an IOMMU is present and
>> > covers all DMA-capable devices, then swiotlb-xen is superfluous.
>>
>> It seems that swiotlb-xen works on this Chromebook since all but two
>> of the DMA capable devices use it when configured with the Kconfig option
>> added here and it seems to work fine so I presume Dom0 is 1:1 mapped as
>> expected. It is possible that on this device, the IOMMU is only covering
>> the two devices that need to use the Exynos IOMMU in the tests I have
>> done.
>> There are many other DMA capable devices that use swiotlb-xen DMA ops
>> on Xen, but I have not checked what DMA ops the other devices use when
>> Linux runs on the Chromebook on bare metal without Xen.
>>
>> So I plan to do some tests and see what DMA ops the other devices use if
>> swiotlb-xen is disabled and also what DMA ops the other devices use when
>> Linux runs on the Chromebook on bare metal without Xen. If these tests
>> show the problem can be fixed by disabling swiotlb-xen with a Kconfig or
>> command line option, I will propose v2 to implement that as a solution.
>>
>> > This last case is the interesting case for virtual IOMMU and Linux
>> usage of
>> > iommu_ops.
>>
>
>
> --
> Mario.
>
I've found a similar error on this post,made on 2012 : https://groups.google.com/g/qubes-devel/c/W1lM4ELuVVI and according to what has been asked there,I want to post some further relevant informations to help you to help me to debug the problem : root@devuan-bunsen:/Dati/xen# xl dmesg ---> https://pastebin.ubuntu.com/p/YvtdCPwMWW/ root@devuan-bunsen:/Dati/xen# dmesg ---> https://pastebin.ubuntu.com/p/9cNxCTXVrd/ root@devuan-bunsen:/var/log/xen# mousepad xenstored-access.log ---> https://pastebin.ubuntu.com/p/RTPBG9nS8R/ root@devuan-bunsen:/var/log/xen# mousepad xenstored.log ---> https://pastebin.ubuntu.com/p/T354ts33nP/ very thanks. On Thu, Nov 16, 2023 at 10:51 AM Mario Marietto <marietto2008@gmail.com> wrote: > Hello to everyone. > > I'm trying to boot Linux 6.1.y as Xen dom0 on the Chromebook xe303c12, aka > Snow and configure and start a very basic domU guest,following the Chuck's > tutorial,located here : > > https://github.com/mobile-virt/u-boot-chromebook-xe303c12/tree/chromebook/xen#starting-a-domu-guest > > What I did has been to carefully follow his instructions,but I haven't > found a solution to fix this problem,yet : > > # sudo xl create devuan.cfg -c > > Parsing config from devuan.cfg > libxl: error: libxl_create.c:720:libxl__domain_make: domain creation fail: > Permission denied > libxl: error: libxl_create.c:1309:initiate_domain_create: cannot make > domain: -3 > > This is my devuan.cfg file : > > kernel = '/Dati/xen/kernels/zImage-6.1.59-stb-xen-cbe+' > memory = '512' > name = 'Devuan' > vcpus = '1' > disk = [ '/Dati/xen/devuan.img,,xvda,w' ] > extra = 'console=hvc0 root=/dev/xvda rw init=/sbin/init > xen-fbfront.video=24,1024,768' > > (I have tried also with root=/dev/xvda1 and root=/dev/xvda2,but leaving > disk = [ '/Dati/xen/devuan.img,,xvda,w' ] and not xvda1 or 2) > > I have no idea about the reason(s) I always get that error,but I don't > think it is caused by a wrong creation of the devuan.img file. Can someone > point me in the right direction to understand what could be wrong ? I > haven't found any useful information on the internet. > > This is bootxen.scr file where I have configured dom0_mem=768 : > > mmc dev 1 > ext2load mmc 1:3 0x42000000 zImage-6.6.0-xen-iommu-dma-on-xen > ext2load mmc 1:3 0x51000000 xen-4.17-armhf-armmp-0x51004000.ub > ext2load mmc 1:3 0x5ffec000 exynos5250-snow.dtb > fdt addr 0x5ffec000 > fdt resize 1024 > fdt set /chosen \#address-cells <0x2> > fdt set /chosen \#size-cells <0x2> > fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 > dom0_mem=768M dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=null" > fdt mknod /chosen dom0 > fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module" > "multiboot,module" > fdt set /chosen/dom0 reg <0x0 0x42000000 0x0 0x87C200 > > fdt set /chosen xen,dom0-bootargs "console=tty1 root=/dev/mmcblk1p4 rw > rootwait clk_ignore_unused --no-log" > bootm 0x51000000 - 0x5ffec000 > > and I've rebooted the Chromebook using this command : > > SMDK5250 # mmc dev 1 > SMDK5250 # ext2load mmc 1:3 0x50000000 bootxen.scr; source 0x50000000 > > > This is the memory available on the machine after having booted the machine ready for xen : > > # free -m > total used free shared buff/cache > available > Mem: 741 329 108 7 332 > 412 > Swap: 0 0 0 > > Thanks in advance for any support. > > On Wed, Nov 15, 2023 at 8:41 PM Mario Marietto <marietto2008@gmail.com> > wrote: > >> ---> So I plan to do some tests and see what DMA ops the other devices >> use if swiotlb-xen is disabled and also what DMA ops the other devices use >> when Linux runs on the Chromebook on bare metal without Xen. If these tests >> show the problem can be fixed by disabling swiotlb-xen with a Kconfig or >> command line option, I will propose v2 to implement that as a solution. >> >> and this could bring you to the next level of our project. Try to install >> xen on different devices. At least it is my next project. I've already >> bought two arm64 phones where xen can be installed because there is a hack >> to overcome the bootloader / hypervisor protection mechanism. For sure I >> hope that you also want to buy them to work on this together. And don't >> worry about how much money they will cost you. I've bought them used and >> refurbished. Or you could buy only one,that I suggest could be the SM-A600G >> (Samsung Galaxy A6) with Exynos7870. Please start looking for it at a good >> price. >> >> On Wed, Nov 15, 2023 at 6:57 PM Chuck Zmudzinski <brchuckz@netscape.net> >> wrote: >> >>> On 11/14/2023 5:20 PM, Stefano Stabellini wrote: >>> > On Tue, 14 Nov 2023, Robin Murphy wrote: >>> >> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote: >>> >> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error >>> when >>> >> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook >>> Snow >>> >> > (and probably on other devices that use the Exynos mixer): >>> >> > >>> >> > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping >>> operations >>> >> > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) >>> >> > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* >>> Device >>> >> > 14450000.mixer lacks support for IOMMU >>> >> > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops >>> 0xc0d97554): -22 >>> >> > exynos-drm exynos-drm: adev bind failed: -22 >>> >> > exynos-dp: probe of 145b0000.dp-controller failed with error -22 >>> >> > >>> >> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when >>> >> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. >>> Enabling >>> >> > the new config option allows devices such as the Exynos mixer to >>> use the >>> >> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the >>> error. >>> >> > >>> >> > The new config option is not set by default because it is likely >>> some >>> >> > devices that use IOMMU for DMA on Xen will cause DMA errors and >>> memory >>> >> > corruption when Xen PV block and network drivers are in use on the >>> system. >>> >> > >>> >> > Link: >>> >> > >>> https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/ >>> >> > >>> >> > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com> >>> >> > --- >>> >> > The reported error with the Exynos mixer is not fixed by default by >>> adding >>> >> > a second patch to select the new option in the Kconfig definition >>> for the >>> >> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it >>> is >>> >> > not certain setting the config option is suitable for all cases. So >>> it is >>> >> > necessary to explicitly select the new config option during the >>> config >>> >> > stage of the Linux kernel build to fix the reported error or similar >>> >> > errors that have the same cause of lack of support for IOMMU on >>> Xen. This >>> >> > is necessary to avoid any regressions that might be caused by >>> enabling the >>> >> > new option by default for the Exynos mixer. >>> >> > arch/arm/mm/dma-mapping.c | 6 ++++++ >>> >> > drivers/xen/Kconfig | 16 ++++++++++++++++ >>> >> > 2 files changed, 22 insertions(+) >>> >> > >>> >> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c >>> >> > index 5409225b4abc..ca04fdf01be3 100644 >>> >> > --- a/arch/arm/mm/dma-mapping.c >>> >> > +++ b/arch/arm/mm/dma-mapping.c >>> >> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, >>> u64 >>> >> > dma_base, u64 size, >>> >> > if (iommu) >>> >> > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, >>> coherent); >>> >> > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN >>> >> >>> >> FWIW I don't think this really needs a config option - if Xen *has* >>> made an >>> >> IOMMU available, then there isn't really much reason not to use it, >>> and if for >>> >> some reason someone really didn't want to then they could simply >>> disable the >>> >> IOMMU driver anyway. >>> > >>> > The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen >>> > doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has >>> > a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on >>> > purpose, it happens by accident. Certain things are going to break, >>> > specifically I am fairly certain PV drivers are going to break. >>> > >>> > If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide >>> it >>> > from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but >>> > doesn't have a driver for.) >>> > >>> > I think it is OK for Chuck and others to play around with this >>> > configuration but I wouldn't add a new kconfig option to Linux to >>> > support it. >>> > >>> > If we do want a kconfig option, I would add a kconfig option or Linux >>> > command line option to enable/disable swiotlb-xen. Basically a way to >>> > force-enable or force-disable xen_swiotlb_detect(). That could be >>> > generally useful for debugging and would also solve the problem here as >>> > it could be used to force-disable swiotlb-xen. I would imagine that the >>> > end result is the same: the default ops (iommu_ops) are used. >>> >>> I will try this. It isn't exactly what I have tested until now because >>> in all my tests so far all the DMA capable devices on the Chromebook use >>> swioltlb-xen except for the two devices that need to use the Exynos IOMMU >>> to fix the error with the Exynos mixer. >>> >>> > >>> > >>> > >>> >> > + if (dev->dma_ops == &iommu_ops) { >>> >> > + dev->archdata.dma_ops_setup = true; >>> >> >>> >> The existing assignment is effectively unconditional by this point >>> anyway, so >>> >> could probably just be moved earlier to save duplicating it (or >>> perhaps just >>> >> make the xen_setup_dma_ops() call conditional instead to save the >>> early return >>> >> as well). >>> >> >>> >> However, are the IOMMU DMA ops really compatible with Xen? The >>> comments about >>> >> hypercalls and foreign memory in xen_arch_need_swiotlb() leave me >>> concerned >>> >> that assuming non-coherent DMA to any old Dom0 page is OK might not >>> actually >>> >> work in general :/ >>> > >>> > Xen has (not yet upstreaming) support for nested IOMMU (Xen uses the >>> > IOMMU while also it exposes a virtual IOMMU to guests.) In those cases >>> > the iommu_ops should be compatible with Xen. >>> > >>> > swiotlb-xen is useful in cases where there is no IOMMU on the platform >>> > (or the IOMMU doesn't cover all DMA-capable devices) and Dom0 is 1:1 >>> > mapped. See include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect. If Dom0 >>> is >>> > not 1:1 mapped swiotlb-xen doesn't work. If an IOMMU is present and >>> > covers all DMA-capable devices, then swiotlb-xen is superfluous. >>> >>> It seems that swiotlb-xen works on this Chromebook since all but two >>> of the DMA capable devices use it when configured with the Kconfig option >>> added here and it seems to work fine so I presume Dom0 is 1:1 mapped as >>> expected. It is possible that on this device, the IOMMU is only covering >>> the two devices that need to use the Exynos IOMMU in the tests I have >>> done. >>> There are many other DMA capable devices that use swiotlb-xen DMA ops >>> on Xen, but I have not checked what DMA ops the other devices use when >>> Linux runs on the Chromebook on bare metal without Xen. >>> >>> So I plan to do some tests and see what DMA ops the other devices use if >>> swiotlb-xen is disabled and also what DMA ops the other devices use when >>> Linux runs on the Chromebook on bare metal without Xen. If these tests >>> show the problem can be fixed by disabling swiotlb-xen with a Kconfig or >>> command line option, I will propose v2 to implement that as a solution. >>> >>> > This last case is the interesting case for virtual IOMMU and Linux >>> usage of >>> > iommu_ops. >>> >> >> >> -- >> Mario. >> > > > -- > Mario. >
My guess is that the issue is one of these two: 1) version mismatch between Xen and Xen tools E.g. Xen from staging and Xen Tools from 4.18 2) missing Xen modules in Linux Dom0 Was /etc/init.d/xencommons run at boot? Is /proc/xen mounted? Are the privcmd, gntdev, xenfs, modules loaded? On Thu, 16 Nov 2023, Mario Marietto wrote: > I've found a similar error on this post,made on 2012 : > https://groups.google.com/g/qubes-devel/c/W1lM4ELuVVI > and according to what has been asked there,I want to post some further relevant informations to help you to help me to debug > the problem : > root@devuan-bunsen:/Dati/xen# xl dmesg ---> https://pastebin.ubuntu.com/p/YvtdCPwMWW/ > > root@devuan-bunsen:/Dati/xen# dmesg ---> https://pastebin.ubuntu.com/p/9cNxCTXVrd/ > root@devuan-bunsen:/var/log/xen# mousepad xenstored-access.log ---> https://pastebin.ubuntu.com/p/RTPBG9nS8R/ > root@devuan-bunsen:/var/log/xen# mousepad xenstored.log ---> https://pastebin.ubuntu.com/p/T354ts33nP/ > > very thanks. > > On Thu, Nov 16, 2023 at 10:51 AM Mario Marietto <marietto2008@gmail.com> wrote: > Hello to everyone. > > I'm trying to boot Linux 6.1.y as Xen dom0 on the Chromebook xe303c12, aka Snow and configure and start a very basic domU > guest,following the Chuck's tutorial,located here : > > https://github.com/mobile-virt/u-boot-chromebook-xe303c12/tree/chromebook/xen#starting-a-domu-guest > > What I did has been to carefully follow his instructions,but I haven't found a solution to fix this problem,yet : > > # sudo xl create devuan.cfg -c > > Parsing config from devuan.cfg > libxl: error: libxl_create.c:720:libxl__domain_make: domain creation fail: Permission denied > libxl: error: libxl_create.c:1309:initiate_domain_create: cannot make domain: -3 > > This is my devuan.cfg file : > > kernel = '/Dati/xen/kernels/zImage-6.1.59-stb-xen-cbe+' > memory = '512' > name = 'Devuan' > vcpus = '1' > disk = [ '/Dati/xen/devuan.img,,xvda,w' ] > extra = 'console=hvc0 root=/dev/xvda rw init=/sbin/init xen-fbfront.video=24,1024,768' > > (I have tried also with root=/dev/xvda1 and root=/dev/xvda2,but leaving disk = [ '/Dati/xen/devuan.img,,xvda,w' ] and not xvda1 or 2) > > I have no idea about the reason(s) I always get that error,but I don't think it is caused by a wrong creation of the devuan.img > file. Can someone point me in the right direction to understand what could be wrong ? I haven't found any useful information on the > internet. > > This is bootxen.scr file where I have configured dom0_mem=768 : > > mmc dev 1 > ext2load mmc 1:3 0x42000000 zImage-6.6.0-xen-iommu-dma-on-xen > ext2load mmc 1:3 0x51000000 xen-4.17-armhf-armmp-0x51004000.ub > ext2load mmc 1:3 0x5ffec000 exynos5250-snow.dtb > fdt addr 0x5ffec000 > fdt resize 1024 > fdt set /chosen \#address-cells <0x2> > fdt set /chosen \#size-cells <0x2> > fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 dom0_mem=768M dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=null" > fdt mknod /chosen dom0 > fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module" "multiboot,module" > fdt set /chosen/dom0 reg <0x0 0x42000000 0x0 0x87C200 > > fdt set /chosen xen,dom0-bootargs "console=tty1 root=/dev/mmcblk1p4 rw rootwait clk_ignore_unused --no-log" > bootm 0x51000000 - 0x5ffec000 > > and I've rebooted the Chromebook using this command : > > SMDK5250 # mmc dev 1 > SMDK5250 # ext2load mmc 1:3 0x50000000 bootxen.scr; source 0x50000000 > > This is the memory available on the machine after having booted the machine ready for xen : > # free -m > total used free shared buff/cache available > Mem: 741 329 108 7 332 412 > Swap: 0 0 0 > > Thanks in advance for any support. > > On Wed, Nov 15, 2023 at 8:41 PM Mario Marietto <marietto2008@gmail.com> wrote: > ---> So I plan to do some tests and see what DMA ops the other devices use if swiotlb-xen is disabled and also what DMA > ops the other devices use when Linux runs on the Chromebook on bare metal without Xen. If these tests show the problem > can be fixed by disabling swiotlb-xen with a Kconfig or command line option, I will propose v2 to implement that as a > solution. > > and this could bring you to the next level of our project. Try to install xen on different devices. At least it is my next > project. I've already bought two arm64 phones where xen can be installed because there is a hack to overcome the bootloader / > hypervisor protection mechanism. For sure I hope that you also want to buy them to work on this together. And don't worry about > how much money they will cost you. I've bought them used and refurbished. Or you could buy only one,that I suggest could be the > SM-A600G (Samsung Galaxy A6) with Exynos7870. Please start looking for it at a good price. > > On Wed, Nov 15, 2023 at 6:57 PM Chuck Zmudzinski <brchuckz@netscape.net> wrote: > On 11/14/2023 5:20 PM, Stefano Stabellini wrote: > > On Tue, 14 Nov 2023, Robin Murphy wrote: > >> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote: > >> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when > >> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow > >> > (and probably on other devices that use the Exynos mixer): > >> > > >> > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations > >> > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) > >> > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device > >> > 14450000.mixer lacks support for IOMMU > >> > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22 > >> > exynos-drm exynos-drm: adev bind failed: -22 > >> > exynos-dp: probe of 145b0000.dp-controller failed with error -22 > >> > > >> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when > >> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling > >> > the new config option allows devices such as the Exynos mixer to use the > >> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error. > >> > > >> > The new config option is not set by default because it is likely some > >> > devices that use IOMMU for DMA on Xen will cause DMA errors and memory > >> > corruption when Xen PV block and network drivers are in use on the system. > >> > > >> > Link: > >> > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/ > >> > > >> > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com> > >> > --- > >> > The reported error with the Exynos mixer is not fixed by default by adding > >> > a second patch to select the new option in the Kconfig definition for the > >> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is > >> > not certain setting the config option is suitable for all cases. So it is > >> > necessary to explicitly select the new config option during the config > >> > stage of the Linux kernel build to fix the reported error or similar > >> > errors that have the same cause of lack of support for IOMMU on Xen. This > >> > is necessary to avoid any regressions that might be caused by enabling the > >> > new option by default for the Exynos mixer. > >> > arch/arm/mm/dma-mapping.c | 6 ++++++ > >> > drivers/xen/Kconfig | 16 ++++++++++++++++ > >> > 2 files changed, 22 insertions(+) > >> > > >> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c > >> > index 5409225b4abc..ca04fdf01be3 100644 > >> > --- a/arch/arm/mm/dma-mapping.c > >> > +++ b/arch/arm/mm/dma-mapping.c > >> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64 > >> > dma_base, u64 size, > >> > if (iommu) > >> > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent); > >> > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN > >> > >> FWIW I don't think this really needs a config option - if Xen *has* made an > >> IOMMU available, then there isn't really much reason not to use it, and if for > >> some reason someone really didn't want to then they could simply disable the > >> IOMMU driver anyway. > > > > The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen > > doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has > > a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on > > purpose, it happens by accident. Certain things are going to break, > > specifically I am fairly certain PV drivers are going to break. > > > > If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it > > from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but > > doesn't have a driver for.) > > > > I think it is OK for Chuck and others to play around with this > > configuration but I wouldn't add a new kconfig option to Linux to > > support it. > > > > If we do want a kconfig option, I would add a kconfig option or Linux > > command line option to enable/disable swiotlb-xen. Basically a way to > > force-enable or force-disable xen_swiotlb_detect(). That could be > > generally useful for debugging and would also solve the problem here as > > it could be used to force-disable swiotlb-xen. I would imagine that the > > end result is the same: the default ops (iommu_ops) are used. > > I will try this. It isn't exactly what I have tested until now because > in all my tests so far all the DMA capable devices on the Chromebook use > swioltlb-xen except for the two devices that need to use the Exynos IOMMU > to fix the error with the Exynos mixer. > > > > > > > > >> > + if (dev->dma_ops == &iommu_ops) { > >> > + dev->archdata.dma_ops_setup = true; > >> > >> The existing assignment is effectively unconditional by this point anyway, so > >> could probably just be moved earlier to save duplicating it (or perhaps just > >> make the xen_setup_dma_ops() call conditional instead to save the early return > >> as well). > >> > >> However, are the IOMMU DMA ops really compatible with Xen? The comments about > >> hypercalls and foreign memory in xen_arch_need_swiotlb() leave me concerned > >> that assuming non-coherent DMA to any old Dom0 page is OK might not actually > >> work in general :/ > > > > Xen has (not yet upstreaming) support for nested IOMMU (Xen uses the > > IOMMU while also it exposes a virtual IOMMU to guests.) In those cases > > the iommu_ops should be compatible with Xen. > > > > swiotlb-xen is useful in cases where there is no IOMMU on the platform > > (or the IOMMU doesn't cover all DMA-capable devices) and Dom0 is 1:1 > > mapped. See include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect. If Dom0 is > > not 1:1 mapped swiotlb-xen doesn't work. If an IOMMU is present and > > covers all DMA-capable devices, then swiotlb-xen is superfluous. > > It seems that swiotlb-xen works on this Chromebook since all but two > of the DMA capable devices use it when configured with the Kconfig option > added here and it seems to work fine so I presume Dom0 is 1:1 mapped as > expected. It is possible that on this device, the IOMMU is only covering > the two devices that need to use the Exynos IOMMU in the tests I have done. > There are many other DMA capable devices that use swiotlb-xen DMA ops > on Xen, but I have not checked what DMA ops the other devices use when > Linux runs on the Chromebook on bare metal without Xen. > > So I plan to do some tests and see what DMA ops the other devices use if > swiotlb-xen is disabled and also what DMA ops the other devices use when > Linux runs on the Chromebook on bare metal without Xen. If these tests > show the problem can be fixed by disabling swiotlb-xen with a Kconfig or > command line option, I will propose v2 to implement that as a solution. > > > This last case is the interesting case for virtual IOMMU and Linux usage of > > iommu_ops. > > > > -- > Mario. > > > > -- > Mario. > > > > -- > Mario. > >
On 11/15/2023 12:56 PM, Chuck Zmudzinski wrote: > On 11/14/2023 5:20 PM, Stefano Stabellini wrote: >> On Tue, 14 Nov 2023, Robin Murphy wrote: >>> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote: >>> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when >>> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow >>> > (and probably on other devices that use the Exynos mixer): >>> > >>> > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations >>> > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) >>> > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device >>> > 14450000.mixer lacks support for IOMMU >>> > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22 >>> > exynos-drm exynos-drm: adev bind failed: -22 >>> > exynos-dp: probe of 145b0000.dp-controller failed with error -22 >>> > >>> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when >>> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling >>> > the new config option allows devices such as the Exynos mixer to use the >>> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error. >>> > >>> > The new config option is not set by default because it is likely some >>> > devices that use IOMMU for DMA on Xen will cause DMA errors and memory >>> > corruption when Xen PV block and network drivers are in use on the system. >>> > >>> > Link: >>> > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/ >>> > >>> > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com> >>> > --- >>> > The reported error with the Exynos mixer is not fixed by default by adding >>> > a second patch to select the new option in the Kconfig definition for the >>> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is >>> > not certain setting the config option is suitable for all cases. So it is >>> > necessary to explicitly select the new config option during the config >>> > stage of the Linux kernel build to fix the reported error or similar >>> > errors that have the same cause of lack of support for IOMMU on Xen. This >>> > is necessary to avoid any regressions that might be caused by enabling the >>> > new option by default for the Exynos mixer. >>> > arch/arm/mm/dma-mapping.c | 6 ++++++ >>> > drivers/xen/Kconfig | 16 ++++++++++++++++ >>> > 2 files changed, 22 insertions(+) >>> > >>> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c >>> > index 5409225b4abc..ca04fdf01be3 100644 >>> > --- a/arch/arm/mm/dma-mapping.c >>> > +++ b/arch/arm/mm/dma-mapping.c >>> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64 >>> > dma_base, u64 size, >>> > if (iommu) >>> > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent); >>> > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN >>> >>> FWIW I don't think this really needs a config option - if Xen *has* made an >>> IOMMU available, then there isn't really much reason not to use it, and if for >>> some reason someone really didn't want to then they could simply disable the >>> IOMMU driver anyway. >> >> The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen >> doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has >> a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on >> purpose, it happens by accident. Certain things are going to break, >> specifically I am fairly certain PV drivers are going to break. >> >> If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it >> from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but >> doesn't have a driver for.) >> >> I think it is OK for Chuck and others to play around with this >> configuration but I wouldn't add a new kconfig option to Linux to >> support it. >> >> If we do want a kconfig option, I would add a kconfig option or Linux >> command line option to enable/disable swiotlb-xen. Basically a way to >> force-enable or force-disable xen_swiotlb_detect(). That could be >> generally useful for debugging and would also solve the problem here as >> it could be used to force-disable swiotlb-xen. I would imagine that the >> end result is the same: the default ops (iommu_ops) are used. Actually, if the swiotlb-xen DMA ops are disabled, arm/mm/dma-mapping.c does not set DMA ops for any of the devices except for the two devices that the Exyos DRM driver uses: the Exyno fimd and the Exynos mixer, and only for those two devices do iommu_ops get set. The same thing happens on bare metal. Only those same two devices have iommu_ops set, and all the other devices do not have any DMA ops set at all on the bare metal, at least not by arm/mm/dma-mapping.c. I will work on implementing the option to disable swiotlb-xen on the command line which makes it possible to do testing and debugging and also enables fixing the problem with the Exynos mixer by setting a command line option instead of recompiling the kernel with a new config option. > > I will try this. It isn't exactly what I have tested until now because > in all my tests so far all the DMA capable devices on the Chromebook use > swioltlb-xen except for the two devices that need to use the Exynos IOMMU > to fix the error with the Exynos mixer. > >> >> >> >>> > + if (dev->dma_ops == &iommu_ops) { >>> > + dev->archdata.dma_ops_setup = true; >>> >>> The existing assignment is effectively unconditional by this point anyway, so >>> could probably just be moved earlier to save duplicating it (or perhaps just >>> make the xen_setup_dma_ops() call conditional instead to save the early return >>> as well). >>> >>> However, are the IOMMU DMA ops really compatible with Xen? The comments about >>> hypercalls and foreign memory in xen_arch_need_swiotlb() leave me concerned >>> that assuming non-coherent DMA to any old Dom0 page is OK might not actually >>> work in general :/ >> >> Xen has (not yet upstreaming) support for nested IOMMU (Xen uses the >> IOMMU while also it exposes a virtual IOMMU to guests.) In those cases >> the iommu_ops should be compatible with Xen. >> >> swiotlb-xen is useful in cases where there is no IOMMU on the platform >> (or the IOMMU doesn't cover all DMA-capable devices) and Dom0 is 1:1 >> mapped. See include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect. If Dom0 is >> not 1:1 mapped swiotlb-xen doesn't work. If an IOMMU is present and >> covers all DMA-capable devices, then swiotlb-xen is superfluous. > > It seems that swiotlb-xen works on this Chromebook since all but two > of the DMA capable devices use it when configured with the Kconfig option > added here and it seems to work fine so I presume Dom0 is 1:1 mapped as > expected. It is possible that on this device, the IOMMU is only covering > the two devices that need to use the Exynos IOMMU in the tests I have done. > There are many other DMA capable devices that use swiotlb-xen DMA ops > on Xen, but I have not checked what DMA ops the other devices use when > Linux runs on the Chromebook on bare metal without Xen. > > So I plan to do some tests and see what DMA ops the other devices use if > swiotlb-xen is disabled and also what DMA ops the other devices use when > Linux runs on the Chromebook on bare metal without Xen. If these tests > show the problem can be fixed by disabling swiotlb-xen with a Kconfig or > command line option, I will propose v2 to implement that as a solution. > >> This last case is the interesting case for virtual IOMMU and Linux usage of >> iommu_ops.
On 11/14/2023 5:20 PM, Stefano Stabellini wrote: > On Tue, 14 Nov 2023, Robin Murphy wrote: >> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote: >> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when >> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow >> > (and probably on other devices that use the Exynos mixer): >> > >> > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations >> > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) >> > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device >> > 14450000.mixer lacks support for IOMMU >> > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22 >> > exynos-drm exynos-drm: adev bind failed: -22 >> > exynos-dp: probe of 145b0000.dp-controller failed with error -22 >> > >> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when >> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling >> > the new config option allows devices such as the Exynos mixer to use the >> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error. >> > >> > The new config option is not set by default because it is likely some >> > devices that use IOMMU for DMA on Xen will cause DMA errors and memory >> > corruption when Xen PV block and network drivers are in use on the system. >> > >> > Link: >> > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/ >> > >> > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com> >> > --- >> > The reported error with the Exynos mixer is not fixed by default by adding >> > a second patch to select the new option in the Kconfig definition for the >> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is >> > not certain setting the config option is suitable for all cases. So it is >> > necessary to explicitly select the new config option during the config >> > stage of the Linux kernel build to fix the reported error or similar >> > errors that have the same cause of lack of support for IOMMU on Xen. This >> > is necessary to avoid any regressions that might be caused by enabling the >> > new option by default for the Exynos mixer. >> > arch/arm/mm/dma-mapping.c | 6 ++++++ >> > drivers/xen/Kconfig | 16 ++++++++++++++++ >> > 2 files changed, 22 insertions(+) >> > >> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c >> > index 5409225b4abc..ca04fdf01be3 100644 >> > --- a/arch/arm/mm/dma-mapping.c >> > +++ b/arch/arm/mm/dma-mapping.c >> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64 >> > dma_base, u64 size, >> > if (iommu) >> > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent); >> > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN >> >> FWIW I don't think this really needs a config option - if Xen *has* made an >> IOMMU available, then there isn't really much reason not to use it, and if for >> some reason someone really didn't want to then they could simply disable the >> IOMMU driver anyway. > > The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen > doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has > a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on > purpose, it happens by accident. Certain things are going to break, > specifically I am fairly certain PV drivers are going to break. > > If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it > from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but > doesn't have a driver for.) > > I think it is OK for Chuck and others to play around with this > configuration but I wouldn't add a new kconfig option to Linux to > support it. > > If we do want a kconfig option, I would add a kconfig option or Linux > command line option to enable/disable swiotlb-xen. Basically a way to > force-enable or force-disable xen_swiotlb_detect().That could be I am trying to understand what you are proposing. I tried disabling the CONFIG_SWIOTLB_XEN option that already exists and it does not seem possible to disable that option without also totally removing Xen dom0 support from Linux on arm. So do you suggest keeping that option as is and adding a Linux command line switch or new Linux Kconfig option that is ignored unless CONFIG_SWIOTLB_XEN is enabled and would make xen_swiotlb_detect() always return false or always return true, depending on the setting of the new option? Thanks, Chuck > generally useful for debugging and would also solve the problem here as > it could be used to force-disable swiotlb-xen. I would imagine that the > end result is the same: the default ops (iommu_ops) are used. >
On Fri, 17 Nov 2023, Fr. Chuck Zmudzinski, C.P.M. wrote: > On 11/14/2023 5:20 PM, Stefano Stabellini wrote: > > On Tue, 14 Nov 2023, Robin Murphy wrote: > >> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote: > >> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when > >> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow > >> > (and probably on other devices that use the Exynos mixer): > >> > > >> > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations > >> > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) > >> > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device > >> > 14450000.mixer lacks support for IOMMU > >> > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22 > >> > exynos-drm exynos-drm: adev bind failed: -22 > >> > exynos-dp: probe of 145b0000.dp-controller failed with error -22 > >> > > >> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when > >> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling > >> > the new config option allows devices such as the Exynos mixer to use the > >> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error. > >> > > >> > The new config option is not set by default because it is likely some > >> > devices that use IOMMU for DMA on Xen will cause DMA errors and memory > >> > corruption when Xen PV block and network drivers are in use on the system. > >> > > >> > Link: > >> > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/ > >> > > >> > Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com> > >> > --- > >> > The reported error with the Exynos mixer is not fixed by default by adding > >> > a second patch to select the new option in the Kconfig definition for the > >> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is > >> > not certain setting the config option is suitable for all cases. So it is > >> > necessary to explicitly select the new config option during the config > >> > stage of the Linux kernel build to fix the reported error or similar > >> > errors that have the same cause of lack of support for IOMMU on Xen. This > >> > is necessary to avoid any regressions that might be caused by enabling the > >> > new option by default for the Exynos mixer. > >> > arch/arm/mm/dma-mapping.c | 6 ++++++ > >> > drivers/xen/Kconfig | 16 ++++++++++++++++ > >> > 2 files changed, 22 insertions(+) > >> > > >> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c > >> > index 5409225b4abc..ca04fdf01be3 100644 > >> > --- a/arch/arm/mm/dma-mapping.c > >> > +++ b/arch/arm/mm/dma-mapping.c > >> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64 > >> > dma_base, u64 size, > >> > if (iommu) > >> > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent); > >> > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN > >> > >> FWIW I don't think this really needs a config option - if Xen *has* made an > >> IOMMU available, then there isn't really much reason not to use it, and if for > >> some reason someone really didn't want to then they could simply disable the > >> IOMMU driver anyway. > > > > The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen > > doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen has > > a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on > > purpose, it happens by accident. Certain things are going to break, > > specifically I am fairly certain PV drivers are going to break. > > > > If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide it > > from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but > > doesn't have a driver for.) > > > > I think it is OK for Chuck and others to play around with this > > configuration but I wouldn't add a new kconfig option to Linux to > > support it. > > > > If we do want a kconfig option, I would add a kconfig option or Linux > > command line option to enable/disable swiotlb-xen. Basically a way to > > force-enable or force-disable xen_swiotlb_detect().That could be > > I am trying to understand what you are proposing. > > I tried disabling the CONFIG_SWIOTLB_XEN option that already > exists and it does not seem possible to disable that option without > also totally removing Xen dom0 support from Linux on arm. So do you > suggest keeping that option as is and adding a Linux command line > switch or new Linux Kconfig option that is ignored unless > CONFIG_SWIOTLB_XEN is enabled and would make xen_swiotlb_detect() > always return false or always return true, depending on the setting > of the new option? Yes. I suggest adding a Linux kernel command line option to force-enable or force-diable swiotlb-xen and that can easily be implemented by changing xen_swiotlb_detect() to return true/false if the command line option is set by the user.
diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 5409225b4abc..ca04fdf01be3 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 size, if (iommu) arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, coherent); +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN + if (dev->dma_ops == &iommu_ops) { + dev->archdata.dma_ops_setup = true; + return; + } +#endif xen_setup_dma_ops(dev); dev->archdata.dma_ops_setup = true; } diff --git a/drivers/xen/Kconfig b/drivers/xen/Kconfig index d5989871dd5d..44e1334b6acd 100644 --- a/drivers/xen/Kconfig +++ b/drivers/xen/Kconfig @@ -181,6 +181,22 @@ config SWIOTLB_XEN select DMA_OPS select SWIOTLB +config ARM_DMA_USE_IOMMU_XEN + bool "Prefer IOMMU DMA ops on Xen" + depends on SWIOTLB_XEN + depends on ARM_DMA_USE_IOMMU + help + Normally on Xen, the IOMMU is used by Xen and not exposed to + Linux. Some Arm systems such as Exynos have an IOMMU that + Xen does not use so the IOMMU is exposed to Linux in those + cases. This option enables Linux to use the IOMMU instead of + using the Xen swiotlb_dma_ops for DMA on Xen. + + Say N here unless support for one or more devices that use + IOMMU ops instead of Xen swiotlb ops for DMA is needed and the + devices that use the IOMMU do not cause any problems on the + Xen system in use. + config XEN_PCI_STUB bool
Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error when attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook Snow (and probably on other devices that use the Exynos mixer): [drm] Exynos DRM: using 14400000.fimd device for DMA mapping operations exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* Device 14450000.mixer lacks support for IOMMU exynos-drm exynos-drm: failed to bind 14450000.mixer (ops 0xc0d97554): -22 exynos-drm exynos-drm: adev bind failed: -22 exynos-dp: probe of 145b0000.dp-controller failed with error -22 Linux normally uses xen_swiotlb_dma_ops for DMA for all devices when xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. Enabling the new config option allows devices such as the Exynos mixer to use the IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the error. The new config option is not set by default because it is likely some devices that use IOMMU for DMA on Xen will cause DMA errors and memory corruption when Xen PV block and network drivers are in use on the system. Link: https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c820@netscape.net/ Signed-off-by: Chuck Zmudzinski <brchuckz@aol.com> --- The reported error with the Exynos mixer is not fixed by default by adding a second patch to select the new option in the Kconfig definition for the Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because it is not certain setting the config option is suitable for all cases. So it is necessary to explicitly select the new config option during the config stage of the Linux kernel build to fix the reported error or similar errors that have the same cause of lack of support for IOMMU on Xen. This is necessary to avoid any regressions that might be caused by enabling the new option by default for the Exynos mixer. arch/arm/mm/dma-mapping.c | 6 ++++++ drivers/xen/Kconfig | 16 ++++++++++++++++ 2 files changed, 22 insertions(+)