[RFC,21/21] arm/cpu-features: Document custom vcpu model

Commit Message

Eric Auger Oct. 25, 2024, 10:17 a.m. UTC

From: Cornelia Huck <[e-mail]>

Add some documentation for the custom model.

Signed-off-by: Eric Auger <[e-mail]>
Signed-off-by: Cornelia Huck <[e-mail]>
---
 docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++-----
 1 file changed, 47 insertions(+), 8 deletions(-)

Comments

Daniel P. Berrangé Oct. 25, 2024, 1:13 p.m. UTC | #1

On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote:
> From: Cornelia Huck <[e-mail]>
> 
> Add some documentation for the custom model.
> 
> Signed-off-by: Eric Auger <[e-mail]>
> Signed-off-by: Cornelia Huck <[e-mail]>
> ---
>  docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++-----
>  1 file changed, 47 insertions(+), 8 deletions(-)


> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU
>  properties have special semantics (see "SVE CPU Property Parsing
>  Semantics").
>  
> +The ``custom`` CPU model needs to be configured via individual ID register
> +field properties, for example::
> +
> +  $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0
> +
> +This forces ID_AA64ISAR0_EL1 DP field to 0.

What is the "baseline" featureset implied by 'custom' ?

On x86 we have the named CPU models each setting a baseline that matches
some corresponding real world silicon. Arm has that too, with TCG at
least. So that way you know what the baseline is that you're toggling
features against.

Experiance on x86 was that making arbitrary feature changes on top of the
named models could often backfire, as there are too many scenarios where
code will check for feature "Y", and assume that existance of "Y" implies
existance of "A", "B" and "C" too. So if you invent custom models where
Y is set, but B is missing, there's decent risk of things going wrong in
horrible to debug ways.  With that in mind, best practice is to try to
just the vanilla named CPU models to the greatest extent possible, and
keep feature toggling to an absolute minimum.  This 'custom' model does
not seem to give us such ability for arm.

With regards,
Daniel

Eric Auger Oct. 25, 2024, 1:28 p.m. UTC | #2

Hi Daniel,

On 10/25/24 15:13, Daniel P. Berrangé wrote:
> On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote:
>> From: Cornelia Huck <[e-mail]>
>>
>> Add some documentation for the custom model.
>>
>> Signed-off-by: Eric Auger <[e-mail]>
>> Signed-off-by: Cornelia Huck <[e-mail]>
>> ---
>>  docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++-----
>>  1 file changed, 47 insertions(+), 8 deletions(-)
>
>> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU
>>  properties have special semantics (see "SVE CPU Property Parsing
>>  Semantics").
>>  
>> +The ``custom`` CPU model needs to be configured via individual ID register
>> +field properties, for example::
>> +
>> +  $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0
>> +
>> +This forces ID_AA64ISAR0_EL1 DP field to 0.
> What is the "baseline" featureset implied by 'custom' ?
there is no baseline at the moment. By default this is a host
passthrough model.
>
> On x86 we have the named CPU models each setting a baseline that matches
> some corresponding real world silicon. Arm has that too, with TCG at
> least. So that way you know what the baseline is that you're toggling
> features against.
Having named models is the next thing. custom vcpu model is not a named
model. But we don't want to TCG like CPU model (like A57) because we
want to be able to migrate between different machines like Ampere to
NVidia or different Ampere systems. So the baseline must be something
usable by both hosts.
>
> Experiance on x86 was that making arbitrary feature changes on top of the
> named models could often backfire, as there are too many scenarios where
> code will check for feature "Y", and assume that existance of "Y" implies
> existance of "A", "B" and "C" too. So if you invent custom models where
> Y is set, but B is missing, there's decent risk of things going wrong in
> horrible to debug ways.  With that in mind, best practice is to try to
> just the vanilla named CPU models to the greatest extent possible, and
> keep feature toggling to an absolute minimum.  This 'custom' model does
> not seem to give us such ability for arm.
The custom model is not yet a named model. This is rather something to
start this kind of discussion.
The code used by the custom vcpu model allows fine tuning of the ID reg
fields. This code could be reused by named models. We can also imagine
that libvirt does implement the named models, ie. hardcodes some IDReg
fields and thus implement the named model instead. Libvirt could
identify what is the baseline source and dest are the closest to, choose
this baseline and tune few reg id fields if some additional tuning are
needed.

Thanks

Eric
>
> With regards,
> Daniel

Daniel P. Berrangé Oct. 25, 2024, 1:31 p.m. UTC | #3

On Fri, Oct 25, 2024 at 03:28:35PM +0200, Eric Auger wrote:
> Hi Daniel,
> 
> On 10/25/24 15:13, Daniel P. Berrangé wrote:
> > On Fri, Oct 25, 2024 at 12:17:40PM +0200, Eric Auger wrote:
> >> From: Cornelia Huck <[e-mail]>
> >>
> >> Add some documentation for the custom model.
> >>
> >> Signed-off-by: Eric Auger <[e-mail]>
> >> Signed-off-by: Cornelia Huck <[e-mail]>
> >> ---
> >>  docs/system/arm/cpu-features.rst | 55 +++++++++++++++++++++++++++-----
> >>  1 file changed, 47 insertions(+), 8 deletions(-)
> >
> >> @@ -167,6 +196,16 @@ disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU
> >>  properties have special semantics (see "SVE CPU Property Parsing
> >>  Semantics").
> >>  
> >> +The ``custom`` CPU model needs to be configured via individual ID register
> >> +field properties, for example::
> >> +
> >> +  $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0
> >> +
> >> +This forces ID_AA64ISAR0_EL1 DP field to 0.
> > What is the "baseline" featureset implied by 'custom' ?
> there is no baseline at the moment. By default this is a host
> passthrough model.

Why do we need to create "custom" at all, as opposed to just letting
users toggle features on "-cpu host" ? 

With regards,
Daniel

Patch

diff --git a/docs/system/arm/cpu-features.rst b/docs/system/arm/cpu-features.rst
index a5fb929243..962a2c6c26 100644
--- a/docs/system/arm/cpu-features.rst
+++ b/docs/system/arm/cpu-features.rst
@@ -2,7 +2,10 @@  Arm CPU Features
 ================
 
 CPU features are optional features that a CPU of supporting type may
-choose to implement or not.  In QEMU, optional CPU features have
+choose to implement or not.  QEMU provides two different mechanisms
+to configure those features:
+
+1. For most CPU models, optional CPU features may have
 corresponding boolean CPU proprieties that, when enabled, indicate
 that the feature is implemented, and, conversely, when disabled,
 indicate that it is not implemented. An example of an Arm CPU feature
@@ -29,6 +32,16 @@  supports the feature.  While ``aarch64`` currently only works with KVM,
 it could work with TCG.  CPU features that are specific to KVM are
 prefixed with "kvm-" and are described in "KVM VCPU Features".
 
+2. Alternatively, the ``custom`` CPU model allows to configure optional
+CPU features via the corresponding ID registers. The host kernel allows
+to write a subset of ID register fields. The custom model exposes
+properties for each write ID register fields. Those options are named
+SYSREG_<IDREG>_<FIELD>. IDREG and FIELD names are those used in the
+ARM ARM Reference Manual. They can also be found in the linux
+arch/arm64/tool/sysreg file which is used to automatically generate the
+description for those registers and fields. The custom model is currently
+only implemented for KVM.
+
 CPU Feature Probing
 ===================
 
@@ -106,6 +119,10 @@  As expected they are now all ``false``.
 
 Only the ``pmu`` CPU feature is available.
 
+Probing for the ``custom`` CPU model is working differently. CPU model
+expansion will return the list of available SYSREG properties (matching
+writable ID register fields)
+
 A note about CPU feature dependencies
 -------------------------------------
 
@@ -119,18 +136,30 @@  independently without error.  For these reasons callers should always
 attempt to make their desired changes all at once in order to ensure the
 collection is valid.
 
+When using the ``custom`` CPU model, the provided set of ID registers
+is always evaluated as a whole.
+
 A note about CPU models and KVM
 -------------------------------
 
 Named CPU models generally do not work with KVM.  There are a few cases
 that do work, e.g. using the named CPU model ``cortex-a57`` with KVM on a
-seattle host, but mostly if KVM is enabled the ``host`` CPU type must be
-used.  This means the guest is provided all the same CPU features as the
-host CPU type has.  And, for this reason, the ``host`` CPU type should
-enable all CPU features that the host has by default.  Indeed it's even
-a bit strange to allow disabling CPU features that the host has when using
-the ``host`` CPU type, but in the absence of CPU models it's the best we can
-do if we want to launch guests without all the host's CPU features enabled.
+seattle host, but mostly if KVM is enabled, either the ``host`` or the
+``custom`` CPU types must be used.
+
+Using the ``host`` type means the guest is provided all the same CPU
+features as the host CPU type has.  And, for this reason, the ``host``
+CPU type should enable all CPU features that the host has by default.
+
+In case some features need to be hidden to the guest, ``custom`` model
+shall be used instead. This is especially useful for migration purpose.
+
+The ``custom`` CPU model generally is the better choice if you want more
+flexibility or stability across different machines or with different kernel
+versions. However, even the ``custom`` CPU model will not allow configuring
+an arbitrary set of features; the ID registers must describe a subset of the
+host's features, and all differences to the host's configuration must actually
+be supported by the kernel to be deconfigured.
 
 Enabling KVM also affects the ``query-cpu-model-expansion`` QMP command.  The
 affect is not only limited to specific features, as pointed out in example
@@ -167,6 +196,16 @@  disabling many SVE vector lengths would be quite verbose, the ``sve<N>`` CPU
 properties have special semantics (see "SVE CPU Property Parsing
 Semantics").
 
+The ``custom`` CPU model needs to be configured via individual ID register
+field properties, for example::
+
+  $ qemu-system-aarch64 -M virt -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0
+
+This forces ID_AA64ISAR0_EL1 DP field to 0.
+
+Note that the other CPU feature properties are not supported when using
+this model.
+
 KVM VCPU Features
 =================