| Message ID | 20240523074040.1611264-10-xin.wang2@amd.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Remaining patches for dynamic node programming using overlay dtbo | expand |
Hi Henry, On 23/05/2024 08:40, Henry Wang wrote: > From: Vikram Garhwal <fnu.vikram@xilinx.com> > > Signed-off-by: Vikram Garhwal <fnu.vikram@xilinx.com> > Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> > Signed-off-by: Henry Wang <xin.wang2@amd.com> > --- > v4: > - No change. > v3: > - No change. > v2: > - Update the content based on the changes in this version. > --- > docs/misc/arm/overlay.txt | 99 +++++++++++++++++++++++++++++++++++++++ > 1 file changed, 99 insertions(+) > create mode 100644 docs/misc/arm/overlay.txt > > diff --git a/docs/misc/arm/overlay.txt b/docs/misc/arm/overlay.txt > new file mode 100644 > index 0000000000..811a6de369 > --- /dev/null > +++ b/docs/misc/arm/overlay.txt > @@ -0,0 +1,99 @@ > +# Device Tree Overlays support in Xen > + > +Xen now supports dynamic device assignment to running domains, This reads as we "support" the feature. I would prefer if we write "Xen expirementally supports..." or similar. > +i.e. adding/removing nodes (using .dtbo) to/from Xen device tree, and > +attaching/detaching them to/from a running domain with given $domid. > + > +Dynamic node assignment works in two steps: > + > +## Add/Remove device tree overlay to/from Xen device tree > + > +1. Xen tools check the dtbo given and parse all other user provided arguments > +2. Xen tools pass the dtbo to Xen hypervisor via hypercall. > +3. Xen hypervisor applies/removes the dtbo to/from Xen device tree. > + > +## Attach/Detach device from the DT overlay to/from domain > + > +1. Xen tools check the dtbo given and parse all other user provided arguments > +2. Xen tools pass the dtbo to Xen hypervisor via hypercall. > +3. Xen hypervisor attach/detach the device to/from the user-provided $domid by > + mapping/unmapping node resources in the DT overlay. > + > +# Examples > + > +Here are a few examples on how to use it. > + > +## Dom0 device add > + > +For assigning a device tree overlay to Dom0, user should firstly properly > +prepare the DT overlay. More information about device tree overlays can be > +found in [1]. Then, in Dom0, enter the following: > + > + (dom0) xl dt-overlay add overlay.dtbo > + > +This will allocate the devices mentioned in overlay.dtbo to Xen device tree. > + > +To assign the newly added device from the dtbo to Dom0: > + > + (dom0) xl dt-overlay attach overlay.dtbo 0 > + > +Next, if the user wants to add the same device tree overlay to dom0 > +Linux, execute the following: > + > + (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay > + (dom0) cat overlay.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo > + > +Finally if needed, the relevant Linux kernel drive can be loaded using: > + > + (dom0) modprobe module_name.ko > + > +## Dom0 device remove > + > +For removing the device from Dom0, first detach the device from Dom0: > + > + (dom0) xl dt-overlay detach overlay.dtbo 0 > + > +NOTE: The user is expected to unload any Linux kernel modules which > +might be accessing the devices in overlay.dtbo before detach the device. > +Detaching devices without unloading the modules might result in a crash. > + > +Then remove the overlay from Xen device tree: > + > + (dom0) xl dt-overlay remove overlay.dtbo > + > +## DomU device add/remove > + > +All the nodes in dtbo will be assigned to a domain; the user will need > +to prepare the dtb for the domU. For example, the `interrupt-parent` property > +of the DomU overlay should be changed to the Xen hardcoded value `0xfde8`. > +Below assumes the properly written DomU dtbo is `overlay_domu.dtbo`. > + > +User will need to create the DomU with below properties properly configured > +in the xl config file: > +- `iomem` I don't quite understand how the user can specify the MMIO region if the device is attached after the domain is created. > +- `passthrough` (if IOMMU is needed) > + > +User will also need to modprobe the relevant drivers. > + > +Example for domU device add: > + > + (dom0) xl dt-overlay add overlay.dtbo # If not executed before > + (dom0) xl dt-overlay attach overlay.dtbo $domid Can how clarify how the MMIO will be mapped? Is it direct mapped? If so, couldn't this result to clash with other part of the address space (e.g. RAM?). > + (dom0) xl console $domid # To access $domid console > + > +Next, if the user needs to modify/prepare the overlay.dtbo suitable for > +the domU: > + > + (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay > + (domU) cat overlay_domu.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo > + > +Finally, if needed, the relevant Linux kernel drive can be probed: > + > + (domU) modprobe module_name.ko > + > +Example for domU overlay remove: > + > + (dom0) xl dt-overlay detach overlay.dtbo $domid > + (dom0) xl dt-overlay remove overlay.dtbo I assume we have safety check in place to ensure we can't remove the device if it is already attached. Is that correct? Cheers,
On Thu, 23 May 2024, Julien Grall wrote: > Hi Henry, > > On 23/05/2024 08:40, Henry Wang wrote: > > From: Vikram Garhwal <fnu.vikram@xilinx.com> > > > > Signed-off-by: Vikram Garhwal <fnu.vikram@xilinx.com> > > Signed-off-by: Stefano Stabellini <stefano.stabellini@xilinx.com> > > Signed-off-by: Henry Wang <xin.wang2@amd.com> > > --- > > v4: > > - No change. > > v3: > > - No change. > > v2: > > - Update the content based on the changes in this version. > > --- > > docs/misc/arm/overlay.txt | 99 +++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 99 insertions(+) > > create mode 100644 docs/misc/arm/overlay.txt > > > > diff --git a/docs/misc/arm/overlay.txt b/docs/misc/arm/overlay.txt > > new file mode 100644 > > index 0000000000..811a6de369 > > --- /dev/null > > +++ b/docs/misc/arm/overlay.txt > > @@ -0,0 +1,99 @@ > > +# Device Tree Overlays support in Xen > > + > > +Xen now supports dynamic device assignment to running domains, > > This reads as we "support" the feature. I would prefer if we write "Xen > expirementally supports..." or similar. Done > > +i.e. adding/removing nodes (using .dtbo) to/from Xen device tree, and > > +attaching/detaching them to/from a running domain with given $domid. > > + > > +Dynamic node assignment works in two steps: > > + > > +## Add/Remove device tree overlay to/from Xen device tree > > + > > +1. Xen tools check the dtbo given and parse all other user provided > > arguments > > +2. Xen tools pass the dtbo to Xen hypervisor via hypercall. > > +3. Xen hypervisor applies/removes the dtbo to/from Xen device tree. > > + > > +## Attach/Detach device from the DT overlay to/from domain > > + > > +1. Xen tools check the dtbo given and parse all other user provided > > arguments > > +2. Xen tools pass the dtbo to Xen hypervisor via hypercall. > > +3. Xen hypervisor attach/detach the device to/from the user-provided $domid > > by > > + mapping/unmapping node resources in the DT overlay. > > + > > +# Examples > > + > > +Here are a few examples on how to use it. > > + > > +## Dom0 device add > > + > > +For assigning a device tree overlay to Dom0, user should firstly properly > > +prepare the DT overlay. More information about device tree overlays can be > > +found in [1]. Then, in Dom0, enter the following: > > + > > + (dom0) xl dt-overlay add overlay.dtbo > > + > > +This will allocate the devices mentioned in overlay.dtbo to Xen device > > tree. > > + > > +To assign the newly added device from the dtbo to Dom0: > > + > > + (dom0) xl dt-overlay attach overlay.dtbo 0 > > + > > +Next, if the user wants to add the same device tree overlay to dom0 > > +Linux, execute the following: > > + > > + (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay > > + (dom0) cat overlay.dtbo > > > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo > > + > > +Finally if needed, the relevant Linux kernel drive can be loaded using: > > + > > + (dom0) modprobe module_name.ko > > + > > +## Dom0 device remove > > + > > +For removing the device from Dom0, first detach the device from Dom0: > > + > > + (dom0) xl dt-overlay detach overlay.dtbo 0 > > + > > +NOTE: The user is expected to unload any Linux kernel modules which > > +might be accessing the devices in overlay.dtbo before detach the device. > > +Detaching devices without unloading the modules might result in a crash. > > + > > +Then remove the overlay from Xen device tree: > > + > > + (dom0) xl dt-overlay remove overlay.dtbo > > + > > +## DomU device add/remove > > + > > +All the nodes in dtbo will be assigned to a domain; the user will need > > +to prepare the dtb for the domU. For example, the `interrupt-parent` > > property > > +of the DomU overlay should be changed to the Xen hardcoded value `0xfde8`. > > +Below assumes the properly written DomU dtbo is `overlay_domu.dtbo`. > > + > > +User will need to create the DomU with below properties properly configured > > +in the xl config file: > > +- `iomem` > > I don't quite understand how the user can specify the MMIO region if the > device is attached after the domain is created. I think this was meant for a domain about to be created (not already running). I clarified. > > > +- `passthrough` (if IOMMU is needed) > > + > > +User will also need to modprobe the relevant drivers. > > + > > +Example for domU device add: > > + > > + (dom0) xl dt-overlay add overlay.dtbo # If not executed > > before > > + (dom0) xl dt-overlay attach overlay.dtbo $domid > > Can how clarify how the MMIO will be mapped? Is it direct mapped? If so, > couldn't this result to clash with other part of the address space (e.g. > RAM?). Yes, it is reusing the same code as dom0, which makes the code nice but it doesn't support non-1:1 mappings. I think those should be done via the xen,reg property. My suggestion would be this: - if xen,reg is present, use it - if xen,reg is not present, fall back to 1:1 mapping based on reg For the next version of the series, I'd just document the current limitation of the implementation. I added this to patch #4: diff --git a/xen/common/dt-overlay.c b/xen/common/dt-overlay.c index 0f8b25ccb4..c2b03865a7 100644 --- a/xen/common/dt-overlay.c +++ b/xen/common/dt-overlay.c @@ -845,7 +845,7 @@ static long handle_attach_overlay_nodes(struct domain *d, uint32_t overlay_fdt_size) { int rc; - unsigned int j; + unsigned int j, len; struct overlay_track *entry; rc = check_overlay_fdt(overlay_fdt, overlay_fdt_size); @@ -888,6 +888,12 @@ static long handle_attach_overlay_nodes(struct domain *d, goto out; } + if ( dt_get_property(overlay_node, "xen,reg", &len) ) + { + printk(XENLOG_ERR "xen,reg not supported yet in overlay\n"); + rc = -EOPNOTSUPP; + goto out; + } write_lock(&dt_host_lock); rc = handle_device(d, overlay_node, p2m_mmio_direct_c, entry->iomem_ranges, entry->irq_ranges); > > + (dom0) xl console $domid # To access $domid > > console > > + > > +Next, if the user needs to modify/prepare the overlay.dtbo suitable for > > +the domU: > > + > > + (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay > > + (domU) cat overlay_domu.dtbo > > > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo > > + > > +Finally, if needed, the relevant Linux kernel drive can be probed: > > + > > + (domU) modprobe module_name.ko > > + > > +Example for domU overlay remove: > > + > > + (dom0) xl dt-overlay detach overlay.dtbo $domid > > + (dom0) xl dt-overlay remove overlay.dtbo > > I assume we have safety check in place to ensure we can't remove the device if > it is already attached. Is that correct? I'll remove this part of the doc.
diff --git a/docs/misc/arm/overlay.txt b/docs/misc/arm/overlay.txt new file mode 100644 index 0000000000..811a6de369 --- /dev/null +++ b/docs/misc/arm/overlay.txt @@ -0,0 +1,99 @@ +# Device Tree Overlays support in Xen + +Xen now supports dynamic device assignment to running domains, +i.e. adding/removing nodes (using .dtbo) to/from Xen device tree, and +attaching/detaching them to/from a running domain with given $domid. + +Dynamic node assignment works in two steps: + +## Add/Remove device tree overlay to/from Xen device tree + +1. Xen tools check the dtbo given and parse all other user provided arguments +2. Xen tools pass the dtbo to Xen hypervisor via hypercall. +3. Xen hypervisor applies/removes the dtbo to/from Xen device tree. + +## Attach/Detach device from the DT overlay to/from domain + +1. Xen tools check the dtbo given and parse all other user provided arguments +2. Xen tools pass the dtbo to Xen hypervisor via hypercall. +3. Xen hypervisor attach/detach the device to/from the user-provided $domid by + mapping/unmapping node resources in the DT overlay. + +# Examples + +Here are a few examples on how to use it. + +## Dom0 device add + +For assigning a device tree overlay to Dom0, user should firstly properly +prepare the DT overlay. More information about device tree overlays can be +found in [1]. Then, in Dom0, enter the following: + + (dom0) xl dt-overlay add overlay.dtbo + +This will allocate the devices mentioned in overlay.dtbo to Xen device tree. + +To assign the newly added device from the dtbo to Dom0: + + (dom0) xl dt-overlay attach overlay.dtbo 0 + +Next, if the user wants to add the same device tree overlay to dom0 +Linux, execute the following: + + (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay + (dom0) cat overlay.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo + +Finally if needed, the relevant Linux kernel drive can be loaded using: + + (dom0) modprobe module_name.ko + +## Dom0 device remove + +For removing the device from Dom0, first detach the device from Dom0: + + (dom0) xl dt-overlay detach overlay.dtbo 0 + +NOTE: The user is expected to unload any Linux kernel modules which +might be accessing the devices in overlay.dtbo before detach the device. +Detaching devices without unloading the modules might result in a crash. + +Then remove the overlay from Xen device tree: + + (dom0) xl dt-overlay remove overlay.dtbo + +## DomU device add/remove + +All the nodes in dtbo will be assigned to a domain; the user will need +to prepare the dtb for the domU. For example, the `interrupt-parent` property +of the DomU overlay should be changed to the Xen hardcoded value `0xfde8`. +Below assumes the properly written DomU dtbo is `overlay_domu.dtbo`. + +User will need to create the DomU with below properties properly configured +in the xl config file: +- `iomem` +- `passthrough` (if IOMMU is needed) + +User will also need to modprobe the relevant drivers. + +Example for domU device add: + + (dom0) xl dt-overlay add overlay.dtbo # If not executed before + (dom0) xl dt-overlay attach overlay.dtbo $domid + (dom0) xl console $domid # To access $domid console + +Next, if the user needs to modify/prepare the overlay.dtbo suitable for +the domU: + + (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay + (domU) cat overlay_domu.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo + +Finally, if needed, the relevant Linux kernel drive can be probed: + + (domU) modprobe module_name.ko + +Example for domU overlay remove: + + (dom0) xl dt-overlay detach overlay.dtbo $domid + (dom0) xl dt-overlay remove overlay.dtbo + +[1] https://www.kernel.org/doc/Documentation/devicetree/overlay-notes.txt