| Message ID | 20230123154735.74832-4-carlo.nonato@minervasys.tech (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Arm cache coloring | expand |
Hi Carlo, On 23/01/2023 16:47, Carlo Nonato wrote: > > > From: Luca Miccio <lucmiccio@gmail.com> > > This commit allows the user to set the cache coloring configuration for > Dom0 via a command line parameter. > Since cache coloring and static memory are incompatible, direct mapping > Dom0 isn't possible when coloring is enabled. > > Here is also introduced a common configuration syntax for cache colors. > > Signed-off-by: Luca Miccio <lucmiccio@gmail.com> > Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> > Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> > --- > v4: > - dom0 colors are dynamically allocated as for any other domain > (colors are duplicated in dom0_colors and in the new array, but logic > is simpler) > --- > docs/misc/arm/cache-coloring.rst | 32 ++++++++++++++++++++++--- > xen/arch/arm/domain_build.c | 17 +++++++++++-- > xen/arch/arm/include/asm/llc_coloring.h | 4 ++++ > xen/arch/arm/llc_coloring.c | 14 +++++++++++ > 4 files changed, 62 insertions(+), 5 deletions(-) > > diff --git a/docs/misc/arm/cache-coloring.rst b/docs/misc/arm/cache-coloring.rst > index 0244d2f606..c2e0e87426 100644 > --- a/docs/misc/arm/cache-coloring.rst > +++ b/docs/misc/arm/cache-coloring.rst > @@ -83,12 +83,38 @@ manually set the way size it's left for the user to overcome failing situations > or for debugging/testing purposes. See `Coloring parameters and domain > configurations`_ section for more information on that. > > +Colors selection format > +*********************** > + > +Regardless of the memory pool that has to be colored (Xen, Dom0/DomUs), > +the color selection can be expressed using the same syntax. In particular a > +comma-separated list of colors or ranges of colors is used. > +Ranges are hyphen-separated intervals (such as `0-4`) and are inclusive on both > +sides. > + > +Note that: > + - no spaces are allowed between values. > + - no overlapping ranges or duplicated colors are allowed. > + - values must be written in ascending order. > + > +Examples: > + > ++---------------------+-----------------------------------+ > +|**Configuration** |**Actual selection** | > ++---------------------+-----------------------------------+ > +| 1-2,5-8 | [1, 2, 5, 6, 7, 8] | > ++---------------------+-----------------------------------+ > +| 4-8,10,11,12 | [4, 5, 6, 7, 8, 10, 11, 12] | > ++---------------------+-----------------------------------+ > +| 0 | [0] | > ++---------------------+-----------------------------------+ > + > Coloring parameters and domain configurations > ********************************************* > > -LLC way size (as previously discussed) can be set using the appropriate command > -line parameter. See the relevant documentation in > -"docs/misc/xen-command-line.pandoc". > +LLC way size (as previously discussed) and Dom0 colors can be set using the > +appropriate command line parameters. See the relevant documentation > +in "docs/misc/xen-command-line.pandoc". > > **Note:** If no color configuration is provided for a domain, the default one, > which corresponds to all available colors, is used instead. > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c > index f35f4d2456..093d4ad6f6 100644 > --- a/xen/arch/arm/domain_build.c > +++ b/xen/arch/arm/domain_build.c > @@ -2,6 +2,7 @@ > #include <xen/init.h> > #include <xen/compile.h> > #include <xen/lib.h> > +#include <xen/llc_coloring.h> > #include <xen/mm.h> > #include <xen/param.h> > #include <xen/domain_page.h> > @@ -4014,7 +4015,10 @@ static int __init construct_dom0(struct domain *d) > /* type must be set before allocate_memory */ > d->arch.type = kinfo.type; > #endif > - allocate_memory_11(d, &kinfo); > + if ( is_domain_llc_colored(d) ) > + allocate_memory(d, &kinfo); While doing some checks, I realized that the issue from previous series is still present. Given that dom0 is hwdom, how are you going to prevent conflicts between allocated RAM and HW resources that are to be mapped to dom0? ~Michal
On 12/01/2024 10:24, Michal Orzel wrote: > > > Hi Carlo, > > On 23/01/2023 16:47, Carlo Nonato wrote: >> >> >> From: Luca Miccio <lucmiccio@gmail.com> >> >> This commit allows the user to set the cache coloring configuration for >> Dom0 via a command line parameter. >> Since cache coloring and static memory are incompatible, direct mapping >> Dom0 isn't possible when coloring is enabled. >> >> Here is also introduced a common configuration syntax for cache colors. >> >> Signed-off-by: Luca Miccio <lucmiccio@gmail.com> >> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> >> Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> >> --- >> v4: >> - dom0 colors are dynamically allocated as for any other domain >> (colors are duplicated in dom0_colors and in the new array, but logic >> is simpler) >> --- >> docs/misc/arm/cache-coloring.rst | 32 ++++++++++++++++++++++--- >> xen/arch/arm/domain_build.c | 17 +++++++++++-- >> xen/arch/arm/include/asm/llc_coloring.h | 4 ++++ >> xen/arch/arm/llc_coloring.c | 14 +++++++++++ >> 4 files changed, 62 insertions(+), 5 deletions(-) >> >> diff --git a/docs/misc/arm/cache-coloring.rst b/docs/misc/arm/cache-coloring.rst >> index 0244d2f606..c2e0e87426 100644 >> --- a/docs/misc/arm/cache-coloring.rst >> +++ b/docs/misc/arm/cache-coloring.rst >> @@ -83,12 +83,38 @@ manually set the way size it's left for the user to overcome failing situations >> or for debugging/testing purposes. See `Coloring parameters and domain >> configurations`_ section for more information on that. >> >> +Colors selection format >> +*********************** >> + >> +Regardless of the memory pool that has to be colored (Xen, Dom0/DomUs), >> +the color selection can be expressed using the same syntax. In particular a >> +comma-separated list of colors or ranges of colors is used. >> +Ranges are hyphen-separated intervals (such as `0-4`) and are inclusive on both >> +sides. >> + >> +Note that: >> + - no spaces are allowed between values. >> + - no overlapping ranges or duplicated colors are allowed. >> + - values must be written in ascending order. >> + >> +Examples: >> + >> ++---------------------+-----------------------------------+ >> +|**Configuration** |**Actual selection** | >> ++---------------------+-----------------------------------+ >> +| 1-2,5-8 | [1, 2, 5, 6, 7, 8] | >> ++---------------------+-----------------------------------+ >> +| 4-8,10,11,12 | [4, 5, 6, 7, 8, 10, 11, 12] | >> ++---------------------+-----------------------------------+ >> +| 0 | [0] | >> ++---------------------+-----------------------------------+ >> + >> Coloring parameters and domain configurations >> ********************************************* >> >> -LLC way size (as previously discussed) can be set using the appropriate command >> -line parameter. See the relevant documentation in >> -"docs/misc/xen-command-line.pandoc". >> +LLC way size (as previously discussed) and Dom0 colors can be set using the >> +appropriate command line parameters. See the relevant documentation >> +in "docs/misc/xen-command-line.pandoc". >> >> **Note:** If no color configuration is provided for a domain, the default one, >> which corresponds to all available colors, is used instead. >> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c >> index f35f4d2456..093d4ad6f6 100644 >> --- a/xen/arch/arm/domain_build.c >> +++ b/xen/arch/arm/domain_build.c >> @@ -2,6 +2,7 @@ >> #include <xen/init.h> >> #include <xen/compile.h> >> #include <xen/lib.h> >> +#include <xen/llc_coloring.h> >> #include <xen/mm.h> >> #include <xen/param.h> >> #include <xen/domain_page.h> >> @@ -4014,7 +4015,10 @@ static int __init construct_dom0(struct domain *d) >> /* type must be set before allocate_memory */ >> d->arch.type = kinfo.type; >> #endif >> - allocate_memory_11(d, &kinfo); >> + if ( is_domain_llc_colored(d) ) >> + allocate_memory(d, &kinfo); > While doing some checks, I realized that the issue from previous series is still present. > Given that dom0 is hwdom, how are you going to prevent conflicts between allocated RAM and HW resources > that are to be mapped to dom0? Sorry. I answered to the wrong revision :) Anyway, the remark still applies to v5. ~Michal
On Fri, 12 Jan 2024, Michal Orzel wrote: > Hi Carlo, > > On 23/01/2023 16:47, Carlo Nonato wrote: > > > > > > From: Luca Miccio <lucmiccio@gmail.com> > > > > This commit allows the user to set the cache coloring configuration for > > Dom0 via a command line parameter. > > Since cache coloring and static memory are incompatible, direct mapping > > Dom0 isn't possible when coloring is enabled. > > > > Here is also introduced a common configuration syntax for cache colors. > > > > Signed-off-by: Luca Miccio <lucmiccio@gmail.com> > > Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> > > Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> > > --- > > v4: > > - dom0 colors are dynamically allocated as for any other domain > > (colors are duplicated in dom0_colors and in the new array, but logic > > is simpler) > > --- > > docs/misc/arm/cache-coloring.rst | 32 ++++++++++++++++++++++--- > > xen/arch/arm/domain_build.c | 17 +++++++++++-- > > xen/arch/arm/include/asm/llc_coloring.h | 4 ++++ > > xen/arch/arm/llc_coloring.c | 14 +++++++++++ > > 4 files changed, 62 insertions(+), 5 deletions(-) > > > > diff --git a/docs/misc/arm/cache-coloring.rst b/docs/misc/arm/cache-coloring.rst > > index 0244d2f606..c2e0e87426 100644 > > --- a/docs/misc/arm/cache-coloring.rst > > +++ b/docs/misc/arm/cache-coloring.rst > > @@ -83,12 +83,38 @@ manually set the way size it's left for the user to overcome failing situations > > or for debugging/testing purposes. See `Coloring parameters and domain > > configurations`_ section for more information on that. > > > > +Colors selection format > > +*********************** > > + > > +Regardless of the memory pool that has to be colored (Xen, Dom0/DomUs), > > +the color selection can be expressed using the same syntax. In particular a > > +comma-separated list of colors or ranges of colors is used. > > +Ranges are hyphen-separated intervals (such as `0-4`) and are inclusive on both > > +sides. > > + > > +Note that: > > + - no spaces are allowed between values. > > + - no overlapping ranges or duplicated colors are allowed. > > + - values must be written in ascending order. > > + > > +Examples: > > + > > ++---------------------+-----------------------------------+ > > +|**Configuration** |**Actual selection** | > > ++---------------------+-----------------------------------+ > > +| 1-2,5-8 | [1, 2, 5, 6, 7, 8] | > > ++---------------------+-----------------------------------+ > > +| 4-8,10,11,12 | [4, 5, 6, 7, 8, 10, 11, 12] | > > ++---------------------+-----------------------------------+ > > +| 0 | [0] | > > ++---------------------+-----------------------------------+ > > + > > Coloring parameters and domain configurations > > ********************************************* > > > > -LLC way size (as previously discussed) can be set using the appropriate command > > -line parameter. See the relevant documentation in > > -"docs/misc/xen-command-line.pandoc". > > +LLC way size (as previously discussed) and Dom0 colors can be set using the > > +appropriate command line parameters. See the relevant documentation > > +in "docs/misc/xen-command-line.pandoc". > > > > **Note:** If no color configuration is provided for a domain, the default one, > > which corresponds to all available colors, is used instead. > > diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c > > index f35f4d2456..093d4ad6f6 100644 > > --- a/xen/arch/arm/domain_build.c > > +++ b/xen/arch/arm/domain_build.c > > @@ -2,6 +2,7 @@ > > #include <xen/init.h> > > #include <xen/compile.h> > > #include <xen/lib.h> > > +#include <xen/llc_coloring.h> > > #include <xen/mm.h> > > #include <xen/param.h> > > #include <xen/domain_page.h> > > @@ -4014,7 +4015,10 @@ static int __init construct_dom0(struct domain *d) > > /* type must be set before allocate_memory */ > > d->arch.type = kinfo.type; > > #endif > > - allocate_memory_11(d, &kinfo); > > + if ( is_domain_llc_colored(d) ) > > + allocate_memory(d, &kinfo); > While doing some checks, I realized that the issue from previous series is still present. > Given that dom0 is hwdom, how are you going to prevent conflicts between allocated RAM and HW resources > that are to be mapped to dom0? Are you referring to the address ranges picked for RAM region and how to make sure they don't conflict with something else (e.g. the MMIO region of a device)? I thought that for dom0 we were reusing the same address layout of the host, so device MMIO would be mapped 1:1, memory would not be mapped 1:1 (due to cache coloring) but it would be mapped to the same guest address ranges corresponding to RAM addresses on the host. Is it not the case in this version of the patch series?
On 18/01/2024 01:23, Stefano Stabellini wrote: > > > On Fri, 12 Jan 2024, Michal Orzel wrote: >> Hi Carlo, >> >> On 23/01/2023 16:47, Carlo Nonato wrote: >>> >>> >>> From: Luca Miccio <lucmiccio@gmail.com> >>> >>> This commit allows the user to set the cache coloring configuration for >>> Dom0 via a command line parameter. >>> Since cache coloring and static memory are incompatible, direct mapping >>> Dom0 isn't possible when coloring is enabled. >>> >>> Here is also introduced a common configuration syntax for cache colors. >>> >>> Signed-off-by: Luca Miccio <lucmiccio@gmail.com> >>> Signed-off-by: Marco Solieri <marco.solieri@minervasys.tech> >>> Signed-off-by: Carlo Nonato <carlo.nonato@minervasys.tech> >>> --- >>> v4: >>> - dom0 colors are dynamically allocated as for any other domain >>> (colors are duplicated in dom0_colors and in the new array, but logic >>> is simpler) >>> --- >>> docs/misc/arm/cache-coloring.rst | 32 ++++++++++++++++++++++--- >>> xen/arch/arm/domain_build.c | 17 +++++++++++-- >>> xen/arch/arm/include/asm/llc_coloring.h | 4 ++++ >>> xen/arch/arm/llc_coloring.c | 14 +++++++++++ >>> 4 files changed, 62 insertions(+), 5 deletions(-) >>> >>> diff --git a/docs/misc/arm/cache-coloring.rst b/docs/misc/arm/cache-coloring.rst >>> index 0244d2f606..c2e0e87426 100644 >>> --- a/docs/misc/arm/cache-coloring.rst >>> +++ b/docs/misc/arm/cache-coloring.rst >>> @@ -83,12 +83,38 @@ manually set the way size it's left for the user to overcome failing situations >>> or for debugging/testing purposes. See `Coloring parameters and domain >>> configurations`_ section for more information on that. >>> >>> +Colors selection format >>> +*********************** >>> + >>> +Regardless of the memory pool that has to be colored (Xen, Dom0/DomUs), >>> +the color selection can be expressed using the same syntax. In particular a >>> +comma-separated list of colors or ranges of colors is used. >>> +Ranges are hyphen-separated intervals (such as `0-4`) and are inclusive on both >>> +sides. >>> + >>> +Note that: >>> + - no spaces are allowed between values. >>> + - no overlapping ranges or duplicated colors are allowed. >>> + - values must be written in ascending order. >>> + >>> +Examples: >>> + >>> ++---------------------+-----------------------------------+ >>> +|**Configuration** |**Actual selection** | >>> ++---------------------+-----------------------------------+ >>> +| 1-2,5-8 | [1, 2, 5, 6, 7, 8] | >>> ++---------------------+-----------------------------------+ >>> +| 4-8,10,11,12 | [4, 5, 6, 7, 8, 10, 11, 12] | >>> ++---------------------+-----------------------------------+ >>> +| 0 | [0] | >>> ++---------------------+-----------------------------------+ >>> + >>> Coloring parameters and domain configurations >>> ********************************************* >>> >>> -LLC way size (as previously discussed) can be set using the appropriate command >>> -line parameter. See the relevant documentation in >>> -"docs/misc/xen-command-line.pandoc". >>> +LLC way size (as previously discussed) and Dom0 colors can be set using the >>> +appropriate command line parameters. See the relevant documentation >>> +in "docs/misc/xen-command-line.pandoc". >>> >>> **Note:** If no color configuration is provided for a domain, the default one, >>> which corresponds to all available colors, is used instead. >>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c >>> index f35f4d2456..093d4ad6f6 100644 >>> --- a/xen/arch/arm/domain_build.c >>> +++ b/xen/arch/arm/domain_build.c >>> @@ -2,6 +2,7 @@ >>> #include <xen/init.h> >>> #include <xen/compile.h> >>> #include <xen/lib.h> >>> +#include <xen/llc_coloring.h> >>> #include <xen/mm.h> >>> #include <xen/param.h> >>> #include <xen/domain_page.h> >>> @@ -4014,7 +4015,10 @@ static int __init construct_dom0(struct domain *d) >>> /* type must be set before allocate_memory */ >>> d->arch.type = kinfo.type; >>> #endif >>> - allocate_memory_11(d, &kinfo); >>> + if ( is_domain_llc_colored(d) ) >>> + allocate_memory(d, &kinfo); >> While doing some checks, I realized that the issue from previous series is still present. >> Given that dom0 is hwdom, how are you going to prevent conflicts between allocated RAM and HW resources >> that are to be mapped to dom0? > > Are you referring to the address ranges picked for RAM region and how to > make sure they don't conflict with something else (e.g. the MMIO region > of a device)? > > I thought that for dom0 we were reusing the same address layout of the > host, so device MMIO would be mapped 1:1, memory would not be mapped 1:1 > (due to cache coloring) but it would be mapped to the same guest address > ranges corresponding to RAM addresses on the host. Is it not the case in > this version of the patch series? It is not the case (see [PATCH v5 03/13] xen/arm: add Dom0 cache coloring support). Hwdom would get assigned the normal Xen guest memory map. Instead we should map memory to host RAM to prevent conflicts with MMIO and reserved memory (e.g. calling find_unallocated_memory in hwdom case to retrieve host RAM excluding rsvm). ~Michal
diff --git a/docs/misc/arm/cache-coloring.rst b/docs/misc/arm/cache-coloring.rst index 0244d2f606..c2e0e87426 100644 --- a/docs/misc/arm/cache-coloring.rst +++ b/docs/misc/arm/cache-coloring.rst @@ -83,12 +83,38 @@ manually set the way size it's left for the user to overcome failing situations or for debugging/testing purposes. See `Coloring parameters and domain configurations`_ section for more information on that. +Colors selection format +*********************** + +Regardless of the memory pool that has to be colored (Xen, Dom0/DomUs), +the color selection can be expressed using the same syntax. In particular a +comma-separated list of colors or ranges of colors is used. +Ranges are hyphen-separated intervals (such as `0-4`) and are inclusive on both +sides. + +Note that: + - no spaces are allowed between values. + - no overlapping ranges or duplicated colors are allowed. + - values must be written in ascending order. + +Examples: + ++---------------------+-----------------------------------+ +|**Configuration** |**Actual selection** | ++---------------------+-----------------------------------+ +| 1-2,5-8 | [1, 2, 5, 6, 7, 8] | ++---------------------+-----------------------------------+ +| 4-8,10,11,12 | [4, 5, 6, 7, 8, 10, 11, 12] | ++---------------------+-----------------------------------+ +| 0 | [0] | ++---------------------+-----------------------------------+ + Coloring parameters and domain configurations ********************************************* -LLC way size (as previously discussed) can be set using the appropriate command -line parameter. See the relevant documentation in -"docs/misc/xen-command-line.pandoc". +LLC way size (as previously discussed) and Dom0 colors can be set using the +appropriate command line parameters. See the relevant documentation +in "docs/misc/xen-command-line.pandoc". **Note:** If no color configuration is provided for a domain, the default one, which corresponds to all available colors, is used instead. diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index f35f4d2456..093d4ad6f6 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -2,6 +2,7 @@ #include <xen/init.h> #include <xen/compile.h> #include <xen/lib.h> +#include <xen/llc_coloring.h> #include <xen/mm.h> #include <xen/param.h> #include <xen/domain_page.h> @@ -4014,7 +4015,10 @@ static int __init construct_dom0(struct domain *d) /* type must be set before allocate_memory */ d->arch.type = kinfo.type; #endif - allocate_memory_11(d, &kinfo); + if ( is_domain_llc_colored(d) ) + allocate_memory(d, &kinfo); + else + allocate_memory_11(d, &kinfo); find_gnttab_region(d, &kinfo); #ifdef CONFIG_STATIC_SHM @@ -4060,6 +4064,8 @@ void __init create_dom0(void) .max_maptrack_frames = -1, .grant_opts = XEN_DOMCTL_GRANT_version(opt_gnttab_max_version), }; + unsigned int *llc_colors = NULL; + unsigned int num_llc_colors = 0, flags = CDF_privileged; /* The vGIC for DOM0 is exactly emulating the hardware GIC */ dom0_cfg.arch.gic_version = XEN_DOMCTL_CONFIG_GIC_NATIVE; @@ -4076,7 +4082,14 @@ void __init create_dom0(void) if ( iommu_enabled ) dom0_cfg.flags |= XEN_DOMCTL_CDF_iommu; - dom0 = domain_create(0, &dom0_cfg, CDF_privileged | CDF_directmap); + if ( llc_coloring_enabled ) + llc_colors = dom0_llc_colors(&num_llc_colors); + else + flags |= CDF_directmap; + + dom0 = domain_create_llc_colored(0, &dom0_cfg, flags, llc_colors, + num_llc_colors); + if ( IS_ERR(dom0) || (alloc_dom0_vcpu0(dom0) == NULL) ) panic("Error creating domain 0\n"); diff --git a/xen/arch/arm/include/asm/llc_coloring.h b/xen/arch/arm/include/asm/llc_coloring.h index c7985c8fd0..382ff7de47 100644 --- a/xen/arch/arm/include/asm/llc_coloring.h +++ b/xen/arch/arm/include/asm/llc_coloring.h @@ -17,9 +17,13 @@ bool __init llc_coloring_init(void); +unsigned int *dom0_llc_colors(unsigned int *num_colors); + #else /* !CONFIG_LLC_COLORING */ static inline bool __init llc_coloring_init(void) { return true; } +static inline unsigned int *dom0_llc_colors( + unsigned int *num_colors) { return NULL; } #endif /* CONFIG_LLC_COLORING */ diff --git a/xen/arch/arm/llc_coloring.c b/xen/arch/arm/llc_coloring.c index 44b601915e..51f057d7c9 100644 --- a/xen/arch/arm/llc_coloring.c +++ b/xen/arch/arm/llc_coloring.c @@ -261,6 +261,20 @@ void domain_dump_llc_colors(struct domain *d) print_colors(d->llc_colors, d->num_llc_colors); } +unsigned int *dom0_llc_colors(unsigned int *num_colors) +{ + unsigned int *colors; + + if ( !dom0_num_colors ) + return NULL; + + colors = alloc_colors(dom0_num_colors); + memcpy(colors, dom0_colors, sizeof(unsigned int) * dom0_num_colors); + *num_colors = dom0_num_colors; + + return colors; +} + /* * Local variables: * mode: C