[00/36] Arm cache coloring

Message

Marco Solieri March 4, 2022, 5:46 p.m. UTC

Shared caches in multi-core CPU architectures represent a problem for
predictability of memory access latency.  This jeopardizes applicability
of many Arm platform in real-time critical and mixed-criticality
scenarios.  We introduce support for cache partitioning with page
coloring, a transparent software technique that enables isolation
between domains and Xen, and thus avoids cache interference.

When creating a domain, a simple syntax (e.g. `0-3` or `4-11`) allows
the user to define assignments of cache partitions ids, called colors,
where assigning different colors guarantees no mutual eviction on cache
will ever happen.  This instructs the Xen memory allocator to provide
the i-th color assignee only with pages that maps to color i, i.e. that
are indexed in the i-th cache partition.

The proposed implementation supports the dom0less experimental feature.
The solution has been tested in several scenarios, including Xilinx Zynq
MPSoCs.


Overview of implementation and commits structure
------------------------------------------------

- Coloring support is added for dom0 and domU by defining the core
  logic, as well as the hardware inspection functionalities used for
  getting needed coloring information [4-17].
- A new memory page allocator that implement the cache coloring
  mechanism is introduced.  The allocation algorithm follows the given
  coloring scheme specified for each domain, and maximizes contiguity in
  the page selection [18-21].
- Coloring support is added to Xen .text region [22-29], as well as to
  dom0less domains [30].
- Extensive documentation details the technique and gently explains
  usage [33-36].


Known limitations
-----------------

- We need to bring back [1-3] the relocation feature in order to move
  Xen memory to a colored space where the hypervisor could be isolated
  from VMs interference.
- When cache coloring is used, static memory assignment is disabled to
  avoid incompatibility. [31]
- Due to assert failure [32], the number of supported colors is
  currently limited at 64, which should be satisfactory for most chips.
  In particular, the problem lies in the cache coloring configuration
  data structure that belongs to each domain.  We are aware that this is
  not a clean solution but we hope that this could be discussed and
  solved within this pull request.


Acknowledgements
----------------

This work is sponsored by Xilinx Inc., and supported by University of
Modena and Reggio Emilia and Minerva Systems.

***

Luca Miccio (36):
  Revert "xen/arm: setup: Add Xen as boot module before printing all
    boot modules"
  Revert "xen/arm: mm: Initialize page-tables earlier"
  xen/arm: restore xen_paddr argument in setup_pagetables
  xen/arm: add parsing function for cache coloring configuration
  xen/arm: compute LLC way size by hardware inspection
  xen/arm: add coloring basic initialization
  xen/arm: add coloring data to domains
  xen/arm: add colored flag to page struct
  xen/arch: add default colors selection function
  xen/arch: check color selection function
  xen/include: define hypercall parameter for coloring
  xen/arm: initialize cache coloring data for Dom0/U
  xen/arm: A domain is not direct mapped when coloring is enabled
  xen/arch: add dump coloring info for domains
  tools: add support for cache coloring configuration
  xen/color alloc: implement color_from_page for ARM64
  xen/arm: add get_max_color function
  Alloc: introduce page_list_for_each_reverse
  xen/arch: introduce cache-coloring allocator
  xen/common: introduce buddy required reservation
  xen/common: add colored allocator initialization
  xen/arch: init cache coloring conf for Xen
  xen/arch: coloring: manually calculate Xen physical addresses
  xen/arm: enable consider_modules for coloring
  xen/arm: bring back get_xen_paddr
  xen/arm: add argument to remove_early_mappings
  xen/arch: add coloring support for Xen
  xen/arm: introduce xen_map_text_rw
  xen/arm: add dump function for coloring info
  xen/arm: add coloring support to dom0less
  Disable coloring if static memory support is selected
  xen/arm: reduce the number of supported colors
  doc, xen-command-line: introduce coloring options
  doc, xl.cfg: introduce coloring configuration option
  doc, device-tree: introduce 'colors' property
  doc, arm: add usage documentation for cache coloring support

 docs/man/xl.cfg.5.pod.in              |  14 +
 docs/misc/arm/cache_coloring.rst      | 191 +++++++++++
 docs/misc/arm/device-tree/booting.txt |   3 +
 docs/misc/xen-command-line.pandoc     |  51 ++-
 tools/libs/light/libxl_arm.c          |  11 +
 tools/libs/light/libxl_types.idl      |   1 +
 tools/xl/xl_parse.c                   |  59 +++-
 xen/arch/arm/Kconfig                  |   6 +
 xen/arch/arm/Makefile                 |   2 +-
 xen/arch/arm/alternative.c            |   8 +-
 xen/arch/arm/coloring.c               | 469 ++++++++++++++++++++++++++
 xen/arch/arm/domain.c                 |  56 +++
 xen/arch/arm/domain_build.c           |  42 ++-
 xen/arch/arm/include/asm/coloring.h   |  98 ++++++
 xen/arch/arm/include/asm/mm.h         |  18 +-
 xen/arch/arm/mm.c                     | 245 +++++++++++++-
 xen/arch/arm/psci.c                   |   4 +-
 xen/arch/arm/setup.c                  |  94 +++++-
 xen/arch/arm/smpboot.c                |  19 +-
 xen/common/page_alloc.c               | 321 +++++++++++++++++-
 xen/common/vmap.c                     |   4 +-
 xen/include/public/arch-arm.h         |   8 +
 xen/include/xen/mm.h                  |   7 +
 xen/include/xen/sched.h               |   4 +
 xen/include/xen/vmap.h                |   2 +
 25 files changed, 1689 insertions(+), 48 deletions(-)
 create mode 100644 docs/misc/arm/cache_coloring.rst
 create mode 100644 xen/arch/arm/coloring.c
 create mode 100644 xen/arch/arm/include/asm/coloring.h