mbox series

[v3,0/9] Arm cache coloring

Message ID 20221022155120.7000-1-carlo.nonato@minervasys.tech (mailing list archive)
Headers show
Series Arm cache coloring | expand

Message

Carlo Nonato Oct. 22, 2022, 3:51 p.m. UTC 
Shared caches in multi-core CPU architectures represent a problem for
predictability of memory access latency. This jeopardizes applicability
of many Arm platform in real-time critical and mixed-criticality
scenarios. We introduce support for cache partitioning with page
coloring, a transparent software technique that enables isolation
between domains and Xen, and thus avoids cache interference.

When creating a domain, a simple syntax (e.g. `0-3` or `4-11`) allows
the user to define assignments of cache partitions ids, called colors,
where assigning different colors guarantees no mutual eviction on cache
will ever happen. This instructs the Xen memory allocator to provide
the i-th color assignee only with pages that maps to color i, i.e. that
are indexed in the i-th cache partition.

The proposed implementation supports the dom0less feature.
The proposed implementation doesn't support the static-mem feature.
The solution has been tested in several scenarios, including Xilinx Zynq
MPSoCs.

Commits structure:
- [1-3] Coloring initialization, cache layout auto-probing and coloring
  data for domains.
- [4-5] xl and Device Tree support for coloring.
- [6] New page allocator for domain memory that implement the cache
  coloring mechanism.
- [7-9] Coloring support for Xen.

Global changes in v3:
- fixed a compilation error because of a forgotten "\"
- replaced some #ifdef with if ( IS_ENABLED )
- other minor changes (docs, typos, variable types, style, etc.)
- better acknowledged Luca Miccio as the original author
- removed #8 since the bootmodule address and size can be replaced without
  the need of this particular revert
- removed #9 since it wasn't a clean revert and thanks to Julien things can
  be done in a smarter way sticking with map_pages_to_xen() (see new #9)

Open points:
- The allocator proposed in #6 works only with order-0 pages and inserts
  them in a sorted list using a linear search. This behavior can be slow if
  large amount of memory is given to it, so the user is warned in the
  documentation for that.
  In a following patch, that I'm going to send separately, a simple buddy
  allocator that indexes pages by color is presented. It can serve higher
  order pages and doesn't need the linear search. Unfortunately, it has
  some flaws that I will discuss there.
- I will address the latest v2 comments from Julien in v4

Acknowledgements
----------------

This work is sponsored by Xilinx Inc., and supported by University of
Modena and Reggio Emilia and Minerva Systems.

Carlo Nonato (6):
  xen/arm: add cache coloring initialization
  xen/arm: add cache coloring initialization for domains
  tools/xl: add support for cache coloring configuration
  xen/arm: add support for cache coloring configuration via device-tree
  Revert "xen/arm: Remove unused BOOT_RELOC_VIRT_START"
  xen/arm: add cache coloring support for Xen

Luca Miccio (3):
  xen/arm: dump cache colors in domain info debug-key
  xen/common: add cache coloring allocator for domains
  xen/arm: add Xen cache colors command line parameter

 docs/man/xl.cfg.5.pod.in              |  10 +
 docs/misc/arm/cache-coloring.rst      | 229 ++++++++++++++
 docs/misc/arm/device-tree/booting.txt |   4 +
 docs/misc/xen-command-line.pandoc     |  49 +++
 tools/libs/light/libxl_create.c       |  12 +
 tools/libs/light/libxl_types.idl      |   1 +
 tools/xl/xl_parse.c                   |  52 +++-
 xen/arch/arm/Kconfig                  |  34 +++
 xen/arch/arm/Makefile                 |   1 +
 xen/arch/arm/alternative.c            |   9 +-
 xen/arch/arm/arm64/head.S             |  48 +++
 xen/arch/arm/coloring.c               | 411 ++++++++++++++++++++++++++
 xen/arch/arm/domain.c                 |   9 +
 xen/arch/arm/domain_build.c           |  26 +-
 xen/arch/arm/include/asm/coloring.h   |  91 ++++++
 xen/arch/arm/include/asm/config.h     |   4 +-
 xen/arch/arm/include/asm/domain.h     |   4 +
 xen/arch/arm/include/asm/mm.h         |  19 +-
 xen/arch/arm/include/asm/processor.h  |  16 +
 xen/arch/arm/mm.c                     |  95 +++++-
 xen/arch/arm/p2m.c                    |   7 +-
 xen/arch/arm/psci.c                   |   4 +-
 xen/arch/arm/setup.c                  |  81 ++++-
 xen/arch/arm/smpboot.c                |   3 +-
 xen/arch/arm/xen.lds.S                |   2 +-
 xen/common/Kconfig                    |   3 +
 xen/common/page_alloc.c               | 259 ++++++++++++++--
 xen/include/public/arch-arm.h         |   8 +
 xen/include/xen/mm.h                  |  43 +++
 29 files changed, 1487 insertions(+), 47 deletions(-)
 create mode 100644 docs/misc/arm/cache-coloring.rst
 create mode 100644 xen/arch/arm/coloring.c
 create mode 100644 xen/arch/arm/include/asm/coloring.h

Comments

Julien Grall Dec. 3, 2022, 5:53 p.m. UTC | #1 
Hi Carlo,

Just in case you are waiting on my review before sending a new version. 
Given that...

On 22/10/2022 16:51, Carlo Nonato wrote:
> Shared caches in multi-core CPU architectures represent a problem for
> predictability of memory access latency. This jeopardizes applicability
> of many Arm platform in real-time critical and mixed-criticality
> scenarios. We introduce support for cache partitioning with page
> coloring, a transparent software technique that enables isolation
> between domains and Xen, and thus avoids cache interference.
> 
> When creating a domain, a simple syntax (e.g. `0-3` or `4-11`) allows
> the user to define assignments of cache partitions ids, called colors,
> where assigning different colors guarantees no mutual eviction on cache
> will ever happen. This instructs the Xen memory allocator to provide
> the i-th color assignee only with pages that maps to color i, i.e. that
> are indexed in the i-th cache partition.
> 
> The proposed implementation supports the dom0less feature.
> The proposed implementation doesn't support the static-mem feature.
> The solution has been tested in several scenarios, including Xilinx Zynq
> MPSoCs.
> 
> Commits structure:
> - [1-3] Coloring initialization, cache layout auto-probing and coloring
>    data for domains.
> - [4-5] xl and Device Tree support for coloring.
> - [6] New page allocator for domain memory that implement the cache
>    coloring mechanism.
> - [7-9] Coloring support for Xen.
> 
> Global changes in v3:
> - fixed a compilation error because of a forgotten "\"
> - replaced some #ifdef with if ( IS_ENABLED )
> - other minor changes (docs, typos, variable types, style, etc.)
> - better acknowledged Luca Miccio as the original author
> - removed #8 since the bootmodule address and size can be replaced without
>    the need of this particular revert
> - removed #9 since it wasn't a clean revert and thanks to Julien things can
>    be done in a smarter way sticking with map_pages_to_xen() (see new #9)
> 
> Open points:
> - The allocator proposed in #6 works only with order-0 pages and inserts
>    them in a sorted list using a linear search. This behavior can be slow if
>    large amount of memory is given to it, so the user is warned in the
>    documentation for that.
>    In a following patch, that I'm going to send separately, a simple buddy
>    allocator that indexes pages by color is presented. It can serve higher
>    order pages and doesn't need the linear search. Unfortunately, it has
>    some flaws that I will discuss there.
> - I will address the latest v2 comments from Julien in v4

... some my comments have not been handled, I am not planning to review 
this version. If there is small parts you want me to have a look before 
you send a new version.

Cheers,