mbox series

[XEN,v9,00/19] dynamic node programming using overlay dtbo

Message ID 20230819002850.32349-1-vikram.garhwal@amd.com (mailing list archive)
Headers show
Series dynamic node programming using overlay dtbo | expand

Message

Vikram Garhwal Aug. 19, 2023, 12:28 a.m. UTC 
Hi,
This patch series is for introducing dynamic programming i.e. add/remove the
devices during run time. Using "xl dt_overlay" a device can be added/removed
with dtbo.

For adding a node using dynamic programming:
    1. flatten device tree overlay node will be added to a fdt
    2. Updated fdt will be unflattened to a new dt_host_new
    3. Extract the newly added node information from dt_host_new
    4. Add the added node under correct parent in original dt_host.
    3. Map/Permit interrupt and iomem region as required.

For removing a node:
    1. Find the node with given path.
    2. Check if the node is used by any of domus. Removes the node only when
        it's not used by any domain.
    3. Removes IRQ permissions and MMIO access.
    5. Find the node in dt_host and delete the device node entry from dt_host.
    6. Free the overlay_tracker entry which means free dt_host_new also(created
in adding node step).

The main purpose of this series to address first part of the dynamic programming
i.e. making Xen aware of new device tree node which means updating the dt_host
with overlay node information. Here we are adding/removing node from dt_host,
and checking/set IOMMU and IRQ permission but never mapping them to any domain.
Right now, mapping/Un-mapping will happen only when a new domU is
created/destroyed using "xl create".

To map IOREQ and IOMMU during runtime, there will be another small series after
this one where we will do the actual IOMMU and IRQ mapping to a running domain
and will call unmap_mmio_regions() to remove the mapping.

Change Log:
 v5 -> v6:
    Add separate patch for memory allocation failure in __unflatten_device_tree().
    Move __unflatten_device_tree() function type changes to single patch.
    Add error propagation for failures in unflatten_dt_node.
    Change CONFIG_OVERLAY_DTB status to "ARM: Tech Preview".
    xen/smmu: Add remove_device callback for smmu_iommu ops:
        Added check to see if device is currently used.
    common/device_tree: Add rwlock for dt_host:
        Addressed feedback from Henry to rearrange code.
    xen/arm: Implement device tree node removal functionalities:
        Changed file name to dash format.
        Addressed Michal's comments.
    Rectified formatting related errors pointed by Michal.

 v4 -> v5:
    Split patch 01/16 to two patches. One with function type changes and another
        with changes inside unflatten_device_tree().
    Change dt_overlay xl command to dt-overlay.
    Protect overlay functionality with CONFIG(arm).
    Fix rwlock issues.
    Move include "device_tree.h" to c file where arch_cpu_init() is called and
        forward declare dt_device_node. This was done to avoid circular deps b/w
        device_tree.h and rwlock.h
    Address Michal's comment on coding style.

 v3 -> v4:
    Add support for adding node's children.
    Add rwlock to dt_host functions.
    Corrected fdt size issue when applying overlay into it.
    Add memory allocation fail handling for unflatten_device_tree().
    Changed xl overlay to xl dt_overlay.
    Correct commit messages.
    Addressed code issue from v3 review.

 v2 -> v3:
    Moved overlay functionalities to dt_overlay.c file.
    Renamed XEN_SYSCTL_overlay to XEN_SYSCTL_dt_overlay.
    Add dt_* prefix to overlay_add/remove_nodes.
    Added dtdevs_lock to protect iommu_add_dt_device().
    For iommu, moved spin_lock to caller.
    Address code issue from v2 review.

 v1 -> v2:
    Add support for multiple node addition/removal using dtbo.
    Replaced fpga-add and fpga-remove with one hypercall overlay_op.
    Moved common domain_build.c function to device.c
    Add OVERLAY_DTB configuration.
    Renamed overlay_get_target() to fdt_overlay_get_target().
    Split remove_device patch into two patches.
    Moved overlay_add/remove code to sysctl and changed it from domctl to sysctl.
    Added all overlay code under CONFIG_OVERLAY_DTB
    Renamed all tool domains fpga function to overlay
    Addressed code issues from v1 review.

Regards,
Vikram

Vikram Garhwal (19):
  common/device_tree: handle memory allocation failure in
    __unflatten_device_tree()
  common/device_tree.c: unflatten_device_tree() propagate errors
  xen/arm/device: Remove __init from function type
  common/device_tree: Export __unflatten_device_tree()
  xen/arm: Add CONFIG_OVERLAY_DTB
  libfdt: Keep fdt functions after init for CONFIG_OVERLAY_DTB.
  libfdt: overlay: change overlay_get_target()
  xen/device-tree: Add device_tree_find_node_by_path() to find nodes in
    device tree
  xen/iommu: Move spin_lock from iommu_dt_device_is_assigned to caller
  xen/iommu: protect iommu_add_dt_device() with dtdevs_lock
  xen/iommu: Introduce iommu_remove_dt_device()
  xen/smmu: Add remove_device callback for smmu_iommu ops
  asm/smp.h: Fix circular dependency for device_tree.h and rwlock.h
  common/device_tree: Add rwlock for dt_host
  xen/arm: Implement device tree node removal functionalities
  xen/arm: Implement device tree node addition functionalities
  tools/libs/ctrl: Implement new xc interfaces for dt overlay
  tools/libs/light: Implement new libxl functions for device tree
    overlay ops
  tools/xl: Add new xl command overlay for device tree overlay support

 CHANGELOG.md                            |    3 +-
 MAINTAINERS                             |    1 +
 SUPPORT.md                              |    6 +
 tools/include/libxl.h                   |   11 +
 tools/include/xenctrl.h                 |    5 +
 tools/libs/ctrl/Makefile.common         |    1 +
 tools/libs/ctrl/xc_dt_overlay.c         |   51 ++
 tools/libs/light/Makefile               |    3 +
 tools/libs/light/libxl_dt_overlay.c     |   71 ++
 tools/xl/xl.h                           |    1 +
 tools/xl/xl_cmdtable.c                  |    6 +
 tools/xl/xl_vmcontrol.c                 |   52 ++
 xen/arch/arm/Kconfig                    |    5 +
 xen/arch/arm/device.c                   |  149 ++++
 xen/arch/arm/domain_build.c             |  147 ----
 xen/arch/arm/include/asm/domain_build.h |    2 -
 xen/arch/arm/include/asm/setup.h        |    6 +
 xen/arch/arm/include/asm/smp.h          |    4 +-
 xen/arch/arm/smpboot.c                  |    1 +
 xen/arch/arm/sysctl.c                   |   16 +-
 xen/common/Makefile                     |    1 +
 xen/common/device_tree.c                |   62 +-
 xen/common/dt-overlay.c                 | 1001 +++++++++++++++++++++++
 xen/common/libfdt/Makefile              |    4 +
 xen/common/libfdt/fdt_overlay.c         |   29 +-
 xen/common/libfdt/version.lds           |    1 +
 xen/drivers/passthrough/arm/smmu.c      |   63 ++
 xen/drivers/passthrough/device_tree.c   |   94 ++-
 xen/include/public/sysctl.h             |   24 +
 xen/include/xen/device_tree.h           |   36 +-
 xen/include/xen/dt-overlay.h            |   63 ++
 xen/include/xen/iommu-private.h         |   28 +
 xen/include/xen/iommu.h                 |    1 +
 xen/include/xen/libfdt/libfdt.h         |   18 +
 34 files changed, 1762 insertions(+), 204 deletions(-)
 create mode 100644 tools/libs/ctrl/xc_dt_overlay.c
 create mode 100644 tools/libs/light/libxl_dt_overlay.c
 create mode 100644 xen/common/dt-overlay.c
 create mode 100644 xen/include/xen/dt-overlay.h
 create mode 100644 xen/include/xen/iommu-private.h

Comments

Julien Grall Aug. 23, 2023, 10:18 p.m. UTC | #1 
On 19/08/2023 01:28, Vikram Garhwal wrote:
> Hi,

Hi Vikram,

> This patch series is for introducing dynamic programming i.e. add/remove the
> devices during run time. Using "xl dt_overlay" a device can be added/removed
> with dtbo.

Do you have any pointer where one can find more details about the dtbo? 
How can it be created? What's the internal device-tree format?

> 
> For adding a node using dynamic programming:
>      1. flatten device tree overlay node will be added to a fdt
>      2. Updated fdt will be unflattened to a new dt_host_new
>      3. Extract the newly added node information from dt_host_new
>      4. Add the added node under correct parent in original dt_host.
>      3. Map/Permit interrupt and iomem region as required.
> 
> For removing a node:
>      1. Find the node with given path.
>      2. Check if the node is used by any of domus. Removes the node only when
>          it's not used by any domain.
>      3. Removes IRQ permissions and MMIO access.
>      5. Find the node in dt_host and delete the device node entry from dt_host.
>      6. Free the overlay_tracker entry which means free dt_host_new also(created
> in adding node step).
> 
> The main purpose of this series to address first part of the dynamic programming
> i.e. making Xen aware of new device tree node which means updating the dt_host
> with overlay node information. Here we are adding/removing node from dt_host,
> and checking/set IOMMU and IRQ permission but never mapping them to any domain.
> Right now, mapping/Un-mapping will happen only when a new domU is
> created/destroyed using "xl create".

I am ok with this restriction. However are you planning to handle it 
because this goes out of tech preview?

The reason I am asking is, AFAICT, your code will not even look at the 
xen,passthrough (which is used to indicate that only permissions will be 
given). So if the ABI becomes stable, then we would need to invent a new 
property to say "Please map the device resources".

I would rather want the behavior of the dt-overlay to act the same as 
for the host DT. IOW, the device would be mapped to dom0 by default 
unless the propery "xen,passthrough" is added.

I would be OK to say that we don't currently support DT overlay if the 
property "xen,passthrough" is not present in the top-level node.

Cheers,