| Message ID | 1477366863-5246-2-git-send-email-yi.y.sun@linux.intel.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
>>> On 25.10.16 at 05:40, <yi.y.sun@linux.intel.com> wrote: > +# References > + > +[Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3, chapter 17.17] > (http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html) Please don't refer to SDM chapters by number. The numbering changes too frequently, while section titles a reasonably stable. Jan
On 16-10-25 07:37:24, Jan Beulich wrote: > >>> On 25.10.16 at 05:40, <yi.y.sun@linux.intel.com> wrote: > > +# References > > + > > +[Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3, chapter 17.17] > > (http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html) > > Please don't refer to SDM chapters by number. The numbering > changes too frequently, while section titles a reasonably stable. > Got it, thanks! > Jan
Hi Yi, On Mon, Oct 24, 2016 at 11:40 PM, Yi Sun <yi.y.sun@linux.intel.com> wrote: > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com> > --- > docs/features/l2_cat.pandoc | 314 ++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 314 insertions(+) > create mode 100644 docs/features/l2_cat.pandoc > > diff --git a/docs/features/l2_cat.pandoc b/docs/features/l2_cat.pandoc > new file mode 100644 > index 0000000..8544510 > --- /dev/null > +++ b/docs/features/l2_cat.pandoc > @@ -0,0 +1,314 @@ > +% Intel L2 Cache Allocation Technology (L2 CAT) Feature > +% Revision 2.0 > + > +\clearpage > + > +# Basics > + > +---------------- ---------------------------------------------------- > + Status: **Tech Preview** > + > +Architecture(s): Intel x86 > + > + Component(s): Hypervisor, toolstack > + > + Hardware: Atom codename Goldmont and beyond > +---------------- ---------------------------------------------------- I'm interested in trying out your code. I'm planning to purchase a SoC with the Atom Goldmont processors. Do you have some suggestions about the SoC I should purchase? I would prefer to use the same SoC as you have. :-) Thank you very much! Meng ----------- Meng Xu PhD Student in Computer and Information Science University of Pennsylvania http://www.cis.upenn.edu/~mengxu/
Hi, Xu, On 16-10-30 11:51:35, Meng Xu wrote: > Hi Yi, > > On Mon, Oct 24, 2016 at 11:40 PM, Yi Sun <yi.y.sun@linux.intel.com> wrote: > > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com> > > --- > > docs/features/l2_cat.pandoc | 314 ++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 314 insertions(+) > > create mode 100644 docs/features/l2_cat.pandoc > > > > +---------------- ---------------------------------------------------- > > + Status: **Tech Preview** > > + > > +Architecture(s): Intel x86 > > + > > + Component(s): Hypervisor, toolstack > > + > > + Hardware: Atom codename Goldmont and beyond > > +---------------- ---------------------------------------------------- > > I'm interested in trying out your code. > I'm planning to purchase a SoC with the Atom Goldmont processors. > Do you have some suggestions about the SoC I should purchase? > I would prefer to use the same SoC as you have. :-) > > Thank you very much! > > Meng > Thank you for being interesting in this feature! The CPU name is Denverton. Thanks, Sun Yi > ----------- > Meng Xu > PhD Student in Computer and Information Science > University of Pennsylvania > http://www.cis.upenn.edu/~mengxu/
On Tue, Oct 25, 2016 at 11:40:49AM +0800, Yi Sun wrote: > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com> > --- > docs/features/l2_cat.pandoc | 314 ++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 314 insertions(+) > create mode 100644 docs/features/l2_cat.pandoc > > diff --git a/docs/features/l2_cat.pandoc b/docs/features/l2_cat.pandoc > new file mode 100644 > index 0000000..8544510 > --- /dev/null > +++ b/docs/features/l2_cat.pandoc > @@ -0,0 +1,314 @@ > +% Intel L2 Cache Allocation Technology (L2 CAT) Feature > +% Revision 2.0 > + > +\clearpage > + > +# Basics > + > +---------------- ---------------------------------------------------- > + Status: **Tech Preview** > + > +Architecture(s): Intel x86 > + > + Component(s): Hypervisor, toolstack > + > + Hardware: Atom codename Goldmont and beyond > +---------------- ---------------------------------------------------- > + > +# Overview > + > +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a Could you define CAT? > +CPU's shared L2 cache based on application priority or Class of Service > +(COS). Each CLOS is configured using capacity bitmasks (CBM) which > +represent cache capacity and indicate the degree of overlap and > +isolation between classes. Once L2 CAT is configured, the processor > +allows access to portions of L2 cache according to the established > +class of service (COS). > + > +# Technical information > + > +L2 CAT is a member of Intel PSR features and part of CAT, it shares Could you define 'PSR' here? Usually when you introduce an acronym you do something like: Intel Problem Solver Resolver (PSR) and that makes it easy for folks to map the acronym to the full feature. > +some base PSR infrastructure in Xen. > + > +## Hardware perspective > + > +L2 CAT defines a new range MSRs to assign different L2 cache access > +patterns which are known as CBMs (Capacity BitMask), each CBM is > +associated with a COS. > + > +``` > + > + +----------------------------+----------------+ > + IA32_PQR_ASSOC | MSR (per socket) | Address | > + +----+---+-------+ +----------------------------+----------------+ > + | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 | > + +----+---+-------+ +----------------------------+----------------+ > + └-------------> | ... | ... | > + +----------------------------+----------------+ > + | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) | > + +----------------------------+----------------+ > +``` > + > +When context switch happens, the COS of VCPU is written to per-thread > +MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation > +according to the corresponding CBM. > + > +## The relationship between L2 CAT and L3 CAT/CDP > + > +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled Could you define CDP? > +while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled. > + > +L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by > +the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing > +patterns from L3 cache. Like L3 CAT/CDP requirement, the bits of CBM of > +L2 CAT must be continuous too. > + > +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same > +associate register `IA32_PQR_ASSOC`, which means one COS associate to a s/COS associate/COS associate's/ ? > +pair of L2 CBM and L3 CBM. > +Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or > +other PSR features in future). In some cases, a VM is permitted to have a > +COS that is beyond one (or more) of PSR features but within the others. > +For instance, let's assume the max COS of L2 CAT is 8 but the max COS of > +L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to > +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no > +limit) since COS 9 is beyond the max COS (8) of L2 CAT. Thanks for the explanation. ..snip.. [didnt' have any questions below]
On 16-11-11 16:33:09, Konrad Rzeszutek Wilk wrote: > On Tue, Oct 25, 2016 at 11:40:49AM +0800, Yi Sun wrote: > > Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com> > > --- > > docs/features/l2_cat.pandoc | 314 ++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 314 insertions(+) > > create mode 100644 docs/features/l2_cat.pandoc > > > > +# Overview > > + > > +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a > > Could you define CAT? > Sure, thanks! > > +# Technical information > > + > > +L2 CAT is a member of Intel PSR features and part of CAT, it shares > > Could you define 'PSR' here? Usually when you introduce an acronym > you do something like: > > Intel Problem Solver Resolver (PSR) > > and that makes it easy for folks to map the acronym to the full feature. > Thanks for tips! > > +## The relationship between L2 CAT and L3 CAT/CDP > > + > > +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled > > Could you define CDP? > Sure, thanks! > > +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same > > +associate register `IA32_PQR_ASSOC`, which means one COS associate to a > > s/COS associate/COS associate's/ ? > Thanks! > > +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no > > +limit) since COS 9 is beyond the max COS (8) of L2 CAT. > > Thanks for the explanation. > > > ..snip.. > [didnt' have any questions below] Thanks a lot for review!
On Fri, 2016-11-11 at 16:33 -0500, Konrad Rzeszutek Wilk wrote: > On Tue, Oct 25, 2016 at 11:40:49AM +0800, Yi Sun wrote: > > --- /dev/null > > +++ b/docs/features/l2_cat.pandoc > > @@ -0,0 +1,314 @@ > > +% Intel L2 Cache Allocation Technology (L2 CAT) Feature > > +% Revision 2.0 > > + > > +\clearpage > > + > > +# Basics > > + > > +---------------- ----------------------------------------------- > > ----- > > + Status: **Tech Preview** > > + > > +Architecture(s): Intel x86 > > + > > + Component(s): Hypervisor, toolstack > > + > > + Hardware: Atom codename Goldmont and beyond > Atom codename Goldmont and beyond CPUs It may sound obvious, but I'd explicitly add that bit. > > +---------------- ----------------------------------------------- > > ----- > > + > > +# Overview > > + > > +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a > > Could you define CAT? > > > > > +CPU's shared L2 cache based on application priority or Class of > > Service > > +(COS). Each CLOS is configured using capacity bitmasks (CBM) which > > +represent cache capacity and indicate the degree of overlap and > > +isolation between classes. Once L2 CAT is configured, the > > processor > > +allows access to portions of L2 cache according to the established > > +class of service (COS). > > + > > +# Technical information > > + > > +L2 CAT is a member of Intel PSR features and part of CAT, it > > shares > > Could you define 'PSR' here? Usually when you introduce an acronym > you do something like: > > Intel Problem Solver Resolver (PSR) > Wasn't it the 'Intel Probabilistic Silicon Reorganizer' ? :-D :-D > and that makes it easy for folks to map the acronym to the full > feature. > Actually, given the density of acronyms, I'd say it would be good to add a "## Terminology" section at the top, and define all of them there upfront. Also, I see that you're meaning this to be committed in tree and act as the L2 CAT feature document. I know that you've been asked to put it in tree (although, the request was for docs/misc/) and I think it's good to have a feature document for L2 CAT. It contains a lot more technical details than the other (few) feature documents we have in tree right now. Personally, I'm fine with that, but I'd say that at least try to filling these sections would be important: (from docs/features/template.pandoc) # Limitations Information concerning incompatibilities with other features or hardware combinations. # Testing Information concerning how to properly test changes affecting this feature. # Areas for improvement List of enhancements which could be undertaken, e.g. to improve the feature itself, or improve interaction with other features. # Known issues List of known issues or bugs. For tech preview or experimental features, this section must contain the list of items needing fixing for its status to be upgraded. Also, it would be really good to have similar documents for the other PSR features we have upstream already (perhaps finding a way for not duplicating all the common information). Thanks and Regards, Dario
A couple of more things about this document... On Tue, 2016-10-25 at 11:40 +0800, Yi Sun wrote: > diff --git a/docs/features/l2_cat.pandoc > The name of the file could be a bit more descriptive. What about something like: intel_psr_l2_cat.pandoc > +## Implementation Description > + > +* Hypervisor interfaces: > + > + 1. Ext: Boot line parameter "psr=cat" now will enable L2 CAT and > L3 > + CAT if hardware supported. > + > + 2. New: SYSCTL: > + - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information. > + > + 3. New: DOMCTL: > + - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a > domain. > + - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a > domain. > + What do these 'Ext' and 'New' prefixes mean? You perhaps what to communicate whether you are extending something that was existing already, or introducing something new... Well, I don't think that is a very valuable piece of information. After all, we want to know that there is a relevant boot parameter, a sysctl and a domctl interface, no matter whether they're new or not (especially considering that this is the first and for now only feature document for PSR). I'd just kill them. > +* xl interfaces: > + > + 1. Ext: psr-cat-show -l2 domain-id > + Show L2 cbm for a domain. > + => XEN_SYSCTL_PSR_CAT_get_l2_info / > + XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM > + > + 2. Ext: psr-mba-set -l2 domain-id cbm > + Set L2 cbm for a domain. > + => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM > + > + 3. Ext: psr-hwinfo > + Show PSR HW information, including L2 CAT > + => XEN_SYSCTL_PSR_CAT_get_l2_info > + Same here. > +# User information > + > +* Feature Enabling: > + > + Add "psr=cat" to boot line parameter to enable all supported level > CAT > + features. > + > +* xl interfaces: > + > + 1. `psr-cat-show [OPTIONS] domain-id`: > + > + Show domain L2 or L3 CAT CBM. > + > + New option `-l` is added. > + `-l2`: Specify cbm for L2 cache. > + `-l3`: Specify cbm for L3 cache. > + > + If neither `-l2` nor `-l3` is given, level 3 is the default > option. > + Sorry for saying this only now, but wouldn't it be more natural, if neither -l2 not -l3 is specified, to show both (or, in general, all that is supported)? > + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`: > + > + Set domain L2 or L3 CBM. > + > + New option `-l` is added. > + `-l2`: Specify cbm for L2 cache. > + `-l3`: Specify cbm for L3 cache. > + > + If neither `-l2` nor `-l3` is given, level 3 is the default > option. > + > + 3. `psr-hwinfo [OPTIONS]`: > + > + Show L2 & L3 CAT HW informations on every socket. > +------------------------------------------------------------------ > ------ > +Date Revision Version Notes > +---------- -------- -------- ------------------------------------- > ------ > +2016-08-12 1.0 Xen 4.7 Initial design > +2016-09-21 2.0 Xen 4.7 Changes according to review comments. > + 1. L2 CBM bits should be continuous > too. > + 2. Describe 'struct feat_info' > + 3. Update 'struct feat_list" > description. > + 4. Update 'struct feat_ops" > description. > + 5. Update `psr-cat-show` description. > + 6. Update `psr-cat-cbm-set` > description. > + 7. Add volume and chapter info in > References. > If you want to keep track of this uncommited version here, I think it's ok. But I'd just cur it short to a quick 1-liner summary and kill the detailed list of changes (which could then be moved in the patch changelog, but below the usual '---' so it won't get commited). Personally, I'd also, avoid specifying any Xen version in for these, and do that starting from the line corresponding to the first version of the document that hits the tree. Like this, it somehow gives the impression that the feature is present in Xen 4.7, which is not true. Regards, Dario
On 16-11-25 18:39:41, Dario Faggioli wrote: > A couple of more things about this document... > > On Tue, 2016-10-25 at 11:40 +0800, Yi Sun wrote: > > diff --git a/docs/features/l2_cat.pandoc > > > The name of the file could be a bit more descriptive. What about > something like: > > intel_psr_l2_cat.pandoc > Thanks for your suggestion! > > +## Implementation Description > > + > > +* Hypervisor interfaces: > > + > > + 1. Ext: Boot line parameter "psr=cat" now will enable L2 CAT and > > L3 > > + CAT if hardware supported. > > + > > + 2. New: SYSCTL: > > + - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information. > > + > > + 3. New: DOMCTL: > > + - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a > > domain. > > + - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a > > domain. > > + > What do these 'Ext' and 'New' prefixes mean? You perhaps what to > communicate whether you are extending something that was existing > already, or introducing something new... Well, I don't think that is a > very valuable piece of information. > > After all, we want to know that there is a relevant boot parameter, a > sysctl and a domctl interface, no matter whether they're new or not > (especially considering that this is the first and for now only feature > document for PSR). > > I'd just kill them. > Yes, your understand is correct. Thanks for your comments! I will kill them. > > +* xl interfaces: > > + > > + 1. Ext: psr-cat-show -l2 domain-id > > + Show L2 cbm for a domain. > > + => XEN_SYSCTL_PSR_CAT_get_l2_info / > > + XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM > > + > > + 2. Ext: psr-mba-set -l2 domain-id cbm > > + Set L2 cbm for a domain. > > + => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM > > + > > + 3. Ext: psr-hwinfo > > + Show PSR HW information, including L2 CAT > > + => XEN_SYSCTL_PSR_CAT_get_l2_info > > + > Same here. > Thank you! > > +# User information > > + > > +* Feature Enabling: > > + > > + Add "psr=cat" to boot line parameter to enable all supported level > > CAT > > + features. > > + > > +* xl interfaces: > > + > > + 1. `psr-cat-show [OPTIONS] domain-id`: > > + > > + Show domain L2 or L3 CAT CBM. > > + > > + New option `-l` is added. > > + `-l2`: Specify cbm for L2 cache. > > + `-l3`: Specify cbm for L3 cache. > > + > > + If neither `-l2` nor `-l3` is given, level 3 is the default > > option. > > + > Sorry for saying this only now, but wouldn't it be more natural, if > neither -l2 not -l3 is specified, to show both (or, in general, all > that is supported)? > This is for backward compatibility. The original command only supports L3 CAT and it does not have '-l' option. But for show command, your suggestion is good. We can show both or prompt user if any one is not supported. > > + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`: > > + > > + Set domain L2 or L3 CBM. > > + > > + New option `-l` is added. > > + `-l2`: Specify cbm for L2 cache. > > + `-l3`: Specify cbm for L3 cache. > > + > > + If neither `-l2` nor `-l3` is given, level 3 is the default > > option. > > + > > + 3. `psr-hwinfo [OPTIONS]`: > > + > > + Show L2 & L3 CAT HW informations on every socket. > > > +------------------------------------------------------------------ > > ------ > > +Date Revision Version Notes > > +---------- -------- -------- ------------------------------------- > > ------ > > +2016-08-12 1.0 Xen 4.7 Initial design > > +2016-09-21 2.0 Xen 4.7 Changes according to review comments. > > + 1. L2 CBM bits should be continuous > > too. > > + 2. Describe 'struct feat_info' > > + 3. Update 'struct feat_list" > > description. > > + 4. Update 'struct feat_ops" > > description. > > + 5. Update `psr-cat-show` description. > > + 6. Update `psr-cat-cbm-set` > > description. > > + 7. Add volume and chapter info in > > References. > > > If you want to keep track of this uncommited version here, I think it's > ok. But I'd just cur it short to a quick 1-liner summary and kill the > detailed list of changes (which could then be moved in the patch > changelog, but below the usual '---' so it won't get commited). > > Personally, I'd also, avoid specifying any Xen version in for these, > and do that starting from the line corresponding to the first version > of the document that hits the tree. Like this, it somehow gives the > impression that the feature is present in Xen 4.7, which is not true. > > Regards, > Dario Thanks a lot for the suggestion! I will remove Xen version and change log here. BRs, Sun Yi > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
On 16-11-25 18:19:17, Dario Faggioli wrote: > On Fri, 2016-11-11 at 16:33 -0500, Konrad Rzeszutek Wilk wrote: > > On Tue, Oct 25, 2016 at 11:40:49AM +0800, Yi Sun wrote: > > > --- /dev/null > > > +++ b/docs/features/l2_cat.pandoc > > > @@ -0,0 +1,314 @@ > > > +% Intel L2 Cache Allocation Technology (L2 CAT) Feature > > > +% Revision 2.0 > > > + > > > +\clearpage > > > + > > > +# Basics > > > + > > > +---------------- ----------------------------------------------- > > > ----- > > > + Status: **Tech Preview** > > > + > > > +Architecture(s): Intel x86 > > > + > > > + Component(s): Hypervisor, toolstack > > > + > > > + Hardware: Atom codename Goldmont and beyond > > > Atom codename Goldmont and beyond CPUs > > It may sound obvious, but I'd explicitly add that bit. > Ok, thanks! > > > +---------------- ----------------------------------------------- > > > ----- > > > + > > > +# Overview > > > + > > > +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a > > > > Could you define CAT? > > > > > > > > +CPU's shared L2 cache based on application priority or Class of > > > Service > > > +(COS). Each CLOS is configured using capacity bitmasks (CBM) which > > > +represent cache capacity and indicate the degree of overlap and > > > +isolation between classes. Once L2 CAT is configured, the > > > processor > > > +allows access to portions of L2 cache according to the established > > > +class of service (COS). > > > + > > > +# Technical information > > > + > > > +L2 CAT is a member of Intel PSR features and part of CAT, it > > > shares > > > > Could you define 'PSR' here? Usually when you introduce an acronym > > you do something like: > > > > Intel Problem Solver Resolver (PSR) > > > Wasn't it the 'Intel Probabilistic Silicon Reorganizer' ? :-D :-D > It is 'Intel Platform Shared Resource' indeed. :-) > > and that makes it easy for folks to map the acronym to the full > > feature. > > > Actually, given the density of acronyms, I'd say it would be good to > add a "## Terminology" section at the top, and define all of them there > upfront. > Thanks! I will add this section. > Also, I see that you're meaning this to be committed in tree and act as > the L2 CAT feature document. I know that you've been asked to put it in > tree (although, the request was for docs/misc/) and I think it's good > to have a feature document for L2 CAT. > > It contains a lot more technical details than the other (few) feature > documents we have in tree right now. Personally, I'm fine with that, > but I'd say that at least try to filling these sections would be > important: > > (from docs/features/template.pandoc) > > # Limitations > Information concerning incompatibilities with other features or > hardware combinations. > > # Testing > Information concerning how to properly test changes affecting this > feature. > > # Areas for improvement > List of enhancements which could be undertaken, e.g. to improve the > feature itself, or improve interaction with other features. > > # Known issues > List of known issues or bugs. For tech preview or experimental > features, this section must contain the list of items needing fixing > for its status to be upgraded. > Thanks! Will try to add these sections. > Also, it would be really good to have similar documents for the other > PSR features we have upstream already (perhaps finding a way for not > duplicating all the common information). > Besides L2 CAT, there are L3 CAT and CDP features for allocation tech. Also, there are CMT/MBM features for monitoring tech. We will discuss these to check if we can add these feature documents step by step. > Thanks and Regards, > Dario > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
On Tue, 2016-11-29 at 12:52 +0800, Yi Sun wrote: > On 16-11-25 18:39:41, Dario Faggioli wrote: > > On Tue, 2016-10-25 at 11:40 +0800, Yi Sun wrote: > > > +* xl interfaces: > > > + > > > + 1. `psr-cat-show [OPTIONS] domain-id`: > > > + > > > + Show domain L2 or L3 CAT CBM. > > > + > > > + New option `-l` is added. > > > + `-l2`: Specify cbm for L2 cache. > > > + `-l3`: Specify cbm for L3 cache. > > > + > > > + If neither `-l2` nor `-l3` is given, level 3 is the default > > > option. > > > + > > Sorry for saying this only now, but wouldn't it be more natural, if > > neither -l2 not -l3 is specified, to show both (or, in general, all > > that is supported)? > > > This is for backward compatibility. The original command only > supports > L3 CAT and it does not have '-l' option. > > But for show command, your suggestion is good. We can show both or > prompt user if any one is not supported. > I don't understand. I was actually talking about 'show' already, basing on this that I see in the patch: + 1. `psr-cat-show [OPTIONS] domain-id`:> > > + + Show domain L2 or L3 CAT CBM. However, now that I look better, I notice that, when explaining the options, you write "Specify cbm for L2 cache.", which is not what I expect show to do. Shouldn't that say "Show" or "Display"? Cut-&-paste error (from below), or there is really something I'm missing? > > > + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`: > > > + > > > + Set domain L2 or L3 CBM. > > > + > > > + New option `-l` is added. > > > + `-l2`: Specify cbm for L2 cache. > > > + `-l3`: Specify cbm for L3 cache. > > > + > > > + If neither `-l2` nor `-l3` is given, level 3 is the default > > > option. > > > + Regards, Dario
On Tue, 2016-11-29 at 13:20 +0800, Yi Sun wrote: > On 16-11-25 18:19:17, Dario Faggioli wrote: > > On Fri, 2016-11-11 at 16:33 -0500, Konrad Rzeszutek Wilk wrote: > > > Intel Problem Solver Resolver (PSR) > > > > > Wasn't it the 'Intel Probabilistic Silicon Reorganizer' ? :-D :-D > > > It is 'Intel Platform Shared Resource' indeed. :-) > Well, that's sad to hear. I'd love Inter to introduce something like a Probabilistic Silicon Reorganizer... That would be a lot of fun! :-) > > Also, it would be really good to have similar documents for the > > other > > PSR features we have upstream already (perhaps finding a way for > > not > > duplicating all the common information). > > > Besides L2 CAT, there are L3 CAT and CDP features for allocation > tech. > Also, there are CMT/MBM features for monitoring tech. > Yep, I know them. > We will discuss > these to check if we can add these feature documents step by step. > That would be awesome! Regards, Dario
On 16-11-29 13:22:41, Dario Faggioli wrote: > On Tue, 2016-11-29 at 12:52 +0800, Yi Sun wrote: > > On 16-11-25 18:39:41, Dario Faggioli wrote: > > > On Tue, 2016-10-25 at 11:40 +0800, Yi Sun wrote: > > > > +* xl interfaces: > > > > + > > > > + 1. `psr-cat-show [OPTIONS] domain-id`: > > > > + > > > > + Show domain L2 or L3 CAT CBM. > > > > + > > > > + New option `-l` is added. > > > > + `-l2`: Specify cbm for L2 cache. > > > > + `-l3`: Specify cbm for L3 cache. > > > > + > > > > + If neither `-l2` nor `-l3` is given, level 3 is the default > > > > option. > > > > + > > > Sorry for saying this only now, but wouldn't it be more natural, if > > > neither -l2 not -l3 is specified, to show both (or, in general, all > > > that is supported)? > > > > > This is for backward compatibility. The original command only > > supports > > L3 CAT and it does not have '-l' option. > > > > But for show command, your suggestion is good. We can show both or > > prompt user if any one is not supported. > > > I don't understand. I was actually talking about 'show' already, basing > on this that I see in the patch: > > + 1. `psr-cat-show [OPTIONS] domain-id`:> > > > + > + Show domain L2 or L3 CAT CBM. > Sorry for confusion. I agree with your suggestion and will try to change codes to show both L2 and L3. > However, now that I look better, I notice that, when explaining the > options, you write "Specify cbm for L2 cache.", which is not what I > expect show to do. Shouldn't that say "Show" or "Display"? > > Cut-&-paste error (from below), or there is really something I'm > missing? > Oh, sorry. It is a Cut-Paste error. Should be 'Show cbm of L2 cache'. Thanks for finding out this error! > > > > + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`: > > > > + > > > > + Set domain L2 or L3 CBM. > > > > + > > > > + New option `-l` is added. > > > > + `-l2`: Specify cbm for L2 cache. > > > > + `-l3`: Specify cbm for L3 cache. > > > > + > > > > + If neither `-l2` nor `-l3` is given, level 3 is the default > > > > option. > > > > + > > Regards, > Dario > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
diff --git a/docs/features/l2_cat.pandoc b/docs/features/l2_cat.pandoc new file mode 100644 index 0000000..8544510 --- /dev/null +++ b/docs/features/l2_cat.pandoc @@ -0,0 +1,314 @@ +% Intel L2 Cache Allocation Technology (L2 CAT) Feature +% Revision 2.0 + +\clearpage + +# Basics + +---------------- ---------------------------------------------------- + Status: **Tech Preview** + +Architecture(s): Intel x86 + + Component(s): Hypervisor, toolstack + + Hardware: Atom codename Goldmont and beyond +---------------- ---------------------------------------------------- + +# Overview + +L2 CAT allows an OS or Hypervisor/VMM to control allocation of a +CPU's shared L2 cache based on application priority or Class of Service +(COS). Each CLOS is configured using capacity bitmasks (CBM) which +represent cache capacity and indicate the degree of overlap and +isolation between classes. Once L2 CAT is configured, the processor +allows access to portions of L2 cache according to the established +class of service (COS). + +# Technical information + +L2 CAT is a member of Intel PSR features and part of CAT, it shares +some base PSR infrastructure in Xen. + +## Hardware perspective + +L2 CAT defines a new range MSRs to assign different L2 cache access +patterns which are known as CBMs (Capacity BitMask), each CBM is +associated with a COS. + +``` + + +----------------------------+----------------+ + IA32_PQR_ASSOC | MSR (per socket) | Address | + +----+---+-------+ +----------------------------+----------------+ + | |COS| | | IA32_L2_QOS_MASK_0 | 0xD10 | + +----+---+-------+ +----------------------------+----------------+ + └-------------> | ... | ... | + +----------------------------+----------------+ + | IA32_L2_QOS_MASK_n | 0xD10+n (n<64) | + +----------------------------+----------------+ +``` + +When context switch happens, the COS of VCPU is written to per-thread +MSR `IA32_PQR_ASSOC`, and then hardware enforces L2 cache allocation +according to the corresponding CBM. + +## The relationship between L2 CAT and L3 CAT/CDP + +L2 CAT is independent of L3 CAT/CDP, which means L2 CAT would be enabled +while L3 CAT/CDP is disabled, or L2 CAT and L3 CAT/CDP are all enabled. + +L2 CAT uses a new range CBMs from 0xD10 ~ 0xD10+n (n<64), following by +the L3 CAT/CDP CBMs, and supports setting different L2 cache accessing +patterns from L3 cache. Like L3 CAT/CDP requirement, the bits of CBM of +L2 CAT must be continuous too. + +N.B. L2 CAT and L3 CAT/CDP share the same COS field in the same +associate register `IA32_PQR_ASSOC`, which means one COS associate to a +pair of L2 CBM and L3 CBM. +Besides, the max COS of L2 CAT may be different from L3 CAT/CDP (or +other PSR features in future). In some cases, a VM is permitted to have a +COS that is beyond one (or more) of PSR features but within the others. +For instance, let's assume the max COS of L2 CAT is 8 but the max COS of +L3 CAT is 16, when a VM is assigned 9 as COS, the L3 CBM associated to +COS 9 would be enforced, but for L2 CAT, the behavior is fully open (no +limit) since COS 9 is beyond the max COS (8) of L2 CAT. + +## Design Overview + +* Core COS/CBM association + + When enforcing L2 CAT, all cores of domains have the same default + COS (COS0) which associated to the fully open CBM (all ones bitmask) + to access all L2 cache. The default COS is used only in hypervisor + and is transparent to tool stack and user. + + System administrator can change PSR allocation policy at runtime by + tool stack. Since L2 CAT share COS with L3 CAT/CDP, a COS corresponds + to a 2-tuple, like [L2 CBM, L3 CBM] with only-CAT enabled, when CDP + is enabled, one COS corresponds to a 3-tuple, like [L2 CBM, + L3 Code_CBM, L3 Data_CBM]. If neither L3 CAT nor L3 CDP is enabled, + things would be easier, one COS corresponds to one L2 CBM. + +* VCPU schedule + + This part reuses L3 CAT COS infrastructure. + +* Multi-sockets + + Different sockets may have different L2 CAT capability (e.g. max COS) + although it is consistent on the same socket. So the capability of + per-socket L2 CAT is specified. + +## Implementation Description + +* Hypervisor interfaces: + + 1. Ext: Boot line parameter "psr=cat" now will enable L2 CAT and L3 + CAT if hardware supported. + + 2. New: SYSCTL: + - XEN_SYSCTL_PSR_CAT_get_l2_info: Get L2 CAT information. + + 3. New: DOMCTL: + - XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM: Get L2 CBM for a domain. + - XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM: Set L2 CBM for a domain. + +* xl interfaces: + + 1. Ext: psr-cat-show -l2 domain-id + Show L2 cbm for a domain. + => XEN_SYSCTL_PSR_CAT_get_l2_info / + XEN_DOMCTL_PSR_CAT_OP_GET_L2_CBM + + 2. Ext: psr-mba-set -l2 domain-id cbm + Set L2 cbm for a domain. + => XEN_DOMCTL_PSR_CAT_OP_SET_L2_CBM + + 3. Ext: psr-hwinfo + Show PSR HW information, including L2 CAT + => XEN_SYSCTL_PSR_CAT_get_l2_info + +* Key data structure: + + 1. Feature HW info + + ``` + struct psr_cat_hw_info { + unsigned int cbm_len; + unsigned int cos_max; + }; + ``` + + - Member `cbm_len` + + `cbm_len` is one of the hardware info of CAT. + + - Member `cos_max` + + `cos_max` is one of the hardware info of CAT. + + 2. Feature list node + + ``` + struct feat_node { + enum psr_feat_type feature; + struct feat_ops ops; + struct psr_cat_hw_info info; + uint64_t cos_reg_val[MAX_COS_REG_NUM]; + struct list_head list; + }; + ``` + + When a PSR enforcement feature is enabled, it will be added into a + feature list. The head of the list is created in psr initialization. + + - Member `feature` + + `feature` is an integer number, to indicate which feature the list entry + corresponds to. + + - Member `ops` + + `ops` maintains a callback function list of the feature. It will be introduced + in details later. + + - Member `info` + + `info` maintains the feature HW information which can be got through + psr_hwinfo command. + + - Member `cos_reg_val` + + `cos_reg_val` is an array to maintain the value set in all COS registers of + the feature. + + 3. Per-socket PSR features information structure + + ``` + struct psr_cat_socket_info { + unsigned int feat_mask; + unsigned int nr_feat; + struct list_head feat_list; + unsigned int cos_ref[MAX_COS_REG_NUM]; + spinlock_t ref_lock; + }; + ``` + + We collect all PSR allocation features information of a socket in + this `struct psr_cat_socket_info`. + + - Member `feat_mask` + + `feat_mask` is a bitmap, to indicate which feature is enabled on + current socket. We define `feat_mask` bitmap as: + + bit 0~1: L3 CAT status, [01] stands for L3 CAT only and [10] + stands for L3 CDP is enalbed. + + bit 2: L2 CAT status. + + - Member `cos_ref` + + `cos_ref` is an array which maintains the reference of one COS. + If the COS is used by one domain, the reference will increase one. + If a domain releases the COS, the reference will decrease one. The + array is indexed by COS. + + 4. Feature operation functions structure + + ``` + struct feat_ops { + void (*init_feature)(unsigned int eax, unsigned int ebx, + unsigned int ecx, unsigned int edx, + struct feat_node *feat, + struct psr_cat_socket_info *info); + int (*get_feat_info)(const struct feat_node *feat, enum cbm_type type, + uint32_t dat[], uint32_t array_len); + int (*get_val)(const struct feat_node *feat, unsigned int cos, + enum cbm_type type, uint64_t *val); + unsigned int (*get_max_cos_max)(const struct feat_node *feat); + unsigned int (*get_cos_num)(const struct feat_node *feat); + int (*get_old_val)(uint64_t val[], + const struct feat_node *feat, + unsigned int old_cos); + int (*set_new_val)(uint64_t val[], + const struct feat_node *feat, + unsigned int old_cos, + enum cbm_type type, + uint64_t m); + int (*compare_val)(const uint64_t val[], const struct feat_node *feat, + unsigned int cos, bool *found); + unsigned int (*get_cos_max_from_type)(const struct feat_node *feat, + enum cbm_type type); + unsigned int (*exceeds_cos_max)(const uint64_t val[], + const struct feat_node *feat, + unsigned int cos); + int (*write_msr)(unsigned int cos, const uint64_t val[], + struct feat_node *feat); + }; + ``` + + We abstract above callback functions to encapsulate the feature specific + behaviors into them. Then, it is easy to add a new feature. We just need: + 1) Implement such ops and callback functions for every feature. + 2) Register the ops into `struct feat_node`. + 3) Add the feature into feature list during CPU initialization. + +# User information + +* Feature Enabling: + + Add "psr=cat" to boot line parameter to enable all supported level CAT + features. + +* xl interfaces: + + 1. `psr-cat-show [OPTIONS] domain-id`: + + Show domain L2 or L3 CAT CBM. + + New option `-l` is added. + `-l2`: Specify cbm for L2 cache. + `-l3`: Specify cbm for L3 cache. + + If neither `-l2` nor `-l3` is given, level 3 is the default option. + + 2. `psr-cat-cbm-set [OPTIONS] domain-id cbm`: + + Set domain L2 or L3 CBM. + + New option `-l` is added. + `-l2`: Specify cbm for L2 cache. + `-l3`: Specify cbm for L3 cache. + + If neither `-l2` nor `-l3` is given, level 3 is the default option. + + 3. `psr-hwinfo [OPTIONS]`: + + Show L2 & L3 CAT HW informations on every socket. + +# References + +[Intel® 64 and IA-32 Architectures Software Developer Manuals, vol3, chapter 17.17](http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html) + +# History + +------------------------------------------------------------------------ +Date Revision Version Notes +---------- -------- -------- ------------------------------------------- +2016-08-12 1.0 Xen 4.7 Initial design +2016-09-21 2.0 Xen 4.7 Changes according to review comments. + 1. L2 CBM bits should be continuous too. + 2. Describe 'struct feat_info' + 3. Update 'struct feat_list" description. + 4. Update 'struct feat_ops" description. + 5. Update `psr-cat-show` description. + 6. Update `psr-cat-cbm-set` description. + 7. Add volume and chapter info in References. +2016-10-24 3.0 Xen 4.7 Changes according to review comments. + 1. Update 'struct feat_list' to 'struct feat_node'. + 2. Update description of 'struct psr_cat_socket_info'. + 3. Update `psr-cat-show` description. + 4. Update `psr-cat-cbm-set` description. +---------- -------- -------- -------------------------------------------
Signed-off-by: Yi Sun <yi.y.sun@linux.intel.com> --- docs/features/l2_cat.pandoc | 314 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 314 insertions(+) create mode 100644 docs/features/l2_cat.pandoc