| Message ID | 20241025101959.601048-1-eric.auger@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | kvm/arm: Introduce a customizable aarch64 KVM host model | expand |
On Fri, Oct 25 2024, Eric Auger <eric.auger@redhat.com> wrote: > This RFC series introduces a KVM host "custom" model. > > Since v6.7 kernel, KVM/arm allows the userspace to overwrite the values > of a subset of ID regs. The list of writable fields continues to grow. > The feature ID range is defined as the AArch64 System register space > with op0==3, op1=={0, 1, 3}, CRn==0, CRm=={0-7}, op2=={0-7}. > > The custom model uses this capability and allows to tune the host > passthrough model by overriding some of the host passthrough ID regs. > > The end goal is to get more flexibility when migrating guests > between different machines. We would like the upper software layer > to be able detect how tunable the vpcu is on both source and destination > and accordingly define a customized KVM host model that can fit > both ends. With the legacy host passthrough model, this migration > use case would fail. > > QEMU queries the host kernel to get the list of writable ID reg > fields and expose all the writable fields as uint64 properties. Those > are named "SYSREG_<REG>_<FIELD>". REG and FIELD names are those > described in ARM ARM Reference manual and linux arch/arm64/tools/sysreg. > Some awk scriptsintroduced in the series help parsing the sysreg file and > generate some code. those scripts are used in a similar way as > scripts/update-linux-headers.sh. In case the ABI gets broken, it is > still possible to manually edit the generated code. However it is > globally expected the REG and FIELD names are stable. > > The list of SYSREG_ID properties can be retrieved through the qmp > monitor using query-cpu-model-expansion [2]. > > The first part of the series mostly consists in migrating id reg > storage from named fields in ARMISARegisters to anonymous index > ordered storage in an IdRegMap struct array. The goal is to have > a generic way to store all id registers, also compatible with the > way we retrieve their writable capability at kernel level through > the KVM_ARM_GET_REG_WRITABLE_MASKS ioctl. Having named fields > prevented us from getting this scalability/genericity. Although the > change is invasive it is quite straightforward and should be easy > to be reviewed. > > Then the bulk of the job is to retrieve the writable ID fields and > match them against a "human readable" description of those fields. > We use awk scripts, derived from kernel arch/arm64/tools/gen-sysreg.awk > (so all the credit to Mark Rutland) that populates a data structure > which describes all the ID regs in sysreg and their fields. We match > writable ID reg fields with those latter and dynamically create a > uint64 property. > > Then we need to extend the list of id regs read from the host > so that we get a chance to let their value overriden and write them > back into KVM . > > This expectation is that this custom KVM host model can prepare for > the advent of named models. Introducing named models with reduced > and explicitly defined features is the next step. > > Obviously this series is not able to cope with non writable ID regs. > For instance the problematic of MIDR/REVIDR setting is not handled > at the moment. > > Connie & Eric > > This series can be found at: > https://github.com/eauger/qemu/tree/custom-cpu-model-rfc > > TESTS: > - with few IDREG fields that can be easily examined from guest > userspace: > -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0,SYSREG_ID_AA64ISAR1_EL1_DPB=0x0 > - migration between custom models > - TCG A57 non regressions. Light testing for TCG though. Deep > review may detect some mistakes when migrating between named fields > and IDRegMap storage > - light testing of introspection. Testing a given writable ID field > value with query-cpu-model-expansion is not supported yet. > > TODO/QUESTIONS: > - Some idreg named fields are not yet migrated to an array storage. > some of them are not in isar struct either. Maybe we could have > handled TCG and KVM separately and it may turn out that this > conversion is unneeded. So as it is quite cumbersome I prefered > to keep it for a later stage. > - the custom model does not come with legacy host properties > such as SVE, MTE, expecially those that induce some KVM > settings. This needs to be fixed. One idea would be to convert these to sugar and configure the relevant features instead. The accelerator-specific properties would need to either stay or be deprecated and later removed, I think; I'd hope there are as few as possible of those. The ones requiring to actively do something (e.g. enable MTE with KVM) are the fun ones. > - The custom model and its exposed properties depend on the host > capabilities. More and more IDREG become writable meaning that > the custom model gains more properties over the time and it is > host linux dependent. At the moment there is no versioning in > place. By default the custom model is a host passthrough model > (besides the legacy functions). So if the end-user tries to set > a field that is not writable from a kernel pov, it will fail. > Nevertheless a versionned custom model could constrain the props > exposed, independently on the host linux capabilities. Alternatively (or additionally?), provide a way to discover the set of writable id regs, e.g. via a qmp command that returns what QEMU got from KVM_ARM_GET_REG_WRITABLE_MASKS. > - the QEMU layer does not take care of IDREG field value consistency. > The kernel neither. I imagine this could be the role of the upper > layer to implement a vcpu profile that makes sure settings are > consistent. Here we come to "named" models. What should they look > like on ARM? I would go with Armv8.6, Armv9.2, etc. that include the mandatory features, and additionally the features that can go on top (base model, can be extended; this is similar to what we already have e.g. for s390 z<whatever> models, but allows for the complication that we have many vendors doing many different designs.) Not sure if we'd also want models for specific cpus, but that could go on top, I guess. > - Implementation details: > - it seems there are a lot of duplications in > the code. ID regs are described in different manners, with different > data structs, for TCG, now for KVM. Base data struct for all accelerators and used for accelerator-agnostic code, accelerator-specific structures on top (so that we can add new accelerators, if wanted?) Once the cpu is up and running, we're not manipulating the regs anyway. > - The IdRegMap->regs is sparsely populated. Maybe a better data > struct could be used, although this is the one chosen for the kernel > uapi. > > References: > > [1] [PATCH v12 00/11] Support writable CPU ID registers from userspace > https://lore.kernel.org/all/20230609190054.1542113-1-oliver.upton@linux.dev/ > > [2] > qemu-system-aarch64 -qmp unix:/home/augere/TEST/QEMU/qmp-sock,server,nowait -M virt --enable-kvm -cpu custom > scripts/qmp/qmp-shell /home/augere/TEST/QEMU/qmp-sock > Welcome to the QMP low-level shell! > Connected to QEMU 9.0.50 > (QEMU) query-cpu-model-expansion type=full model={"name":"custom"} > > [3] > KVM_CAP_ARM_SUPPORTED_REG_MASK_RANGES > KVM_ARM_GET_REG_WRITABLE_MASKS > Documentation/virt/kvm/api.rst > > [4] linux "sysreg" file > linux/arch/arm64/tools/sysreg and gen-sysreg.awk > ./tools/include/generated/asm/sysreg-defs.h > > > Cornelia Huck (4): > kvm: kvm_get_writable_id_regs > virt: Allow custom vcpu model in arm virt > arm-qmp-cmds: introspection for custom model > arm/cpu-features: Document custom vcpu model > > Eric Auger (17): > arm/cpu: Add sysreg definitions in cpu-sysegs.h > arm/cpu: Store aa64isar0 into the idregs arrays > arm/cpu: Store aa64isar1/2 into the idregs array > arm/cpu: Store aa64drf0/1 into the idregs array > arm/cpu: Store aa64mmfr0-3 into the idregs array > arm/cpu: Store aa64drf0/1 into the idregs array > arm/cpu: Store aa64smfr0 into the idregs array > arm/cpu: Store id_isar0-7 into the idregs array > arm/cpu: Store id_mfr0/1 into the idregs array > arm/cpu: Store id_dfr0/1 into the idregs array > arm/cpu: Store id_mmfr0-5 into the idregs array > arm/cpu: Add infra to handle generated ID register definitions > arm/cpu: Add sysreg generation scripts > arm/cpu: Add generated files > arm/kvm: Allow reading all the writable ID registers > arm/kvm: write back modified ID regs to KVM > arm/cpu: Introduce a customizable kvm host cpu model > > docs/system/arm/cpu-features.rst | 55 ++- > target/arm/cpu-custom.h | 58 +++ > target/arm/cpu-features.h | 307 ++++++------ > target/arm/cpu-sysregs.h | 152 ++++++ > target/arm/cpu.h | 120 +++-- > target/arm/internals.h | 6 +- > target/arm/kvm_arm.h | 16 +- > hw/arm/virt.c | 3 + > hw/intc/armv7m_nvic.c | 27 +- > target/arm/arm-qmp-cmds.c | 56 ++- > target/arm/cpu-sysreg-properties.c | 682 ++++++++++++++++++++++++++ > target/arm/cpu.c | 124 +++-- > target/arm/cpu64.c | 265 +++++++--- > target/arm/helper.c | 68 +-- > target/arm/kvm.c | 253 +++++++--- > target/arm/ptw.c | 6 +- > target/arm/tcg/cpu-v7m.c | 174 +++---- > target/arm/tcg/cpu32.c | 320 ++++++------ > target/arm/tcg/cpu64.c | 460 ++++++++--------- > scripts/gen-cpu-sysreg-properties.awk | 325 ++++++++++++ > scripts/gen-cpu-sysregs-header.awk | 47 ++ > scripts/update-aarch64-sysreg-code.sh | 27 + > target/arm/meson.build | 1 + > target/arm/trace-events | 8 + > 24 files changed, 2646 insertions(+), 914 deletions(-) > create mode 100644 target/arm/cpu-custom.h > create mode 100644 target/arm/cpu-sysregs.h > create mode 100644 target/arm/cpu-sysreg-properties.c > create mode 100755 scripts/gen-cpu-sysreg-properties.awk > create mode 100755 scripts/gen-cpu-sysregs-header.awk > create mode 100755 scripts/update-aarch64-sysreg-code.sh > > -- > 2.41.0
On Fri, Oct 25, 2024 at 12:17:19PM +0200, Eric Auger wrote: Hi Eric, I'm new to Arm, so please bear with my questions :) > This RFC series introduces a KVM host "custom" model. (a) On terminology: as we know, in the x86 world, QEMU uses these terms[1]: - Host passthrough - Named CPU models - Then there's the libvirt abstraction, "host-model", that aims to provide the best of 'host-passthrough' + named CPU models. Now I see the term "host 'custom' model" here. Most management-layer tools and libvirt users are familiar with the classic terms "host-model" or "custom". If we now say "host 'custom' model", it can create confusion. I hope we can settle on one of the existing terms, or create a new term if need be. (I'll share one more thought on how layers above libvirt tend to use the term "custom", as a reply to patch 21/21, "arm/cpu-features: Document custom vcpu model".) (b) The current CPU features doc[2] for Arm doesn't mention "host passthrough" at all. It is only implied by the last part of this paragraph, from the section titled "A note about CPU models and KVM"[3]: "Named CPU models generally do not work with KVM. There are a few cases that do work [...] but mostly if KVM is enabled the 'host' CPU type must be used." Related: in your reply[4] to Dan in this series, you write: "Having named models is the next thing". So named CPU models will be a thing in Arm, too? Then the above statement in the Arm 'cpu-features' will need updating :-) [...] > - the QEMU layer does not take care of IDREG field value consistency. > The kernel neither. I imagine this could be the role of the upper > layer to implement a vcpu profile that makes sure settings are > consistent. Here we come to "named" models. What should they look > like on ARM? Are there reasons why they can't be similar to how x86 reports in `qemu-system-x86 -cpu help`? E.g. If it's an NVIDIA "Grace A02" (Neoverse-V2) host, it can report: [gracehopper] $> qemu-kvm -cpu help Available CPUs: gracehopper-neoverse-v2 cortex-a57 (deprecated) host max Or whatever is the preferred nomenclature for ARM. It also gives users of both x86 and ARM deployments a consistent expectation. Currently on a "Grace A02" ("Neoverse-V2") machine, it reports: [gracehopper] $> qemu-kvm -cpu help Available CPUs: cortex-a57 (deprecated) host max I see it's because there are no named models yet on ARM :-) [...] [1] https://www.qemu.org/docs/master/system/i386/cpu.html [2] https://www.qemu.org/docs/master/system/arm/cpu-features.html [3] https://www.qemu.org/docs/master/system/arm/cpu-features.html#a-note-about-cpu-models-and-kvm [4] https://lists.nongnu.org/archive/html/qemu-arm/2024-10/msg00891.html
This RFC series introduces a KVM host "custom" model. Since v6.7 kernel, KVM/arm allows the userspace to overwrite the values of a subset of ID regs. The list of writable fields continues to grow. The feature ID range is defined as the AArch64 System register space with op0==3, op1=={0, 1, 3}, CRn==0, CRm=={0-7}, op2=={0-7}. The custom model uses this capability and allows to tune the host passthrough model by overriding some of the host passthrough ID regs. The end goal is to get more flexibility when migrating guests between different machines. We would like the upper software layer to be able detect how tunable the vpcu is on both source and destination and accordingly define a customized KVM host model that can fit both ends. With the legacy host passthrough model, this migration use case would fail. QEMU queries the host kernel to get the list of writable ID reg fields and expose all the writable fields as uint64 properties. Those are named "SYSREG_<REG>_<FIELD>". REG and FIELD names are those described in ARM ARM Reference manual and linux arch/arm64/tools/sysreg. Some awk scriptsintroduced in the series help parsing the sysreg file and generate some code. those scripts are used in a similar way as scripts/update-linux-headers.sh. In case the ABI gets broken, it is still possible to manually edit the generated code. However it is globally expected the REG and FIELD names are stable. The list of SYSREG_ID properties can be retrieved through the qmp monitor using query-cpu-model-expansion [2]. The first part of the series mostly consists in migrating id reg storage from named fields in ARMISARegisters to anonymous index ordered storage in an IdRegMap struct array. The goal is to have a generic way to store all id registers, also compatible with the way we retrieve their writable capability at kernel level through the KVM_ARM_GET_REG_WRITABLE_MASKS ioctl. Having named fields prevented us from getting this scalability/genericity. Although the change is invasive it is quite straightforward and should be easy to be reviewed. Then the bulk of the job is to retrieve the writable ID fields and match them against a "human readable" description of those fields. We use awk scripts, derived from kernel arch/arm64/tools/gen-sysreg.awk (so all the credit to Mark Rutland) that populates a data structure which describes all the ID regs in sysreg and their fields. We match writable ID reg fields with those latter and dynamically create a uint64 property. Then we need to extend the list of id regs read from the host so that we get a chance to let their value overriden and write them back into KVM . This expectation is that this custom KVM host model can prepare for the advent of named models. Introducing named models with reduced and explicitly defined features is the next step. Obviously this series is not able to cope with non writable ID regs. For instance the problematic of MIDR/REVIDR setting is not handled at the moment. Connie & Eric This series can be found at: https://github.com/eauger/qemu/tree/custom-cpu-model-rfc TESTS: - with few IDREG fields that can be easily examined from guest userspace: -cpu custom,SYSREG_ID_AA64ISAR0_EL1_DP=0x0,SYSREG_ID_AA64ISAR1_EL1_DPB=0x0 - migration between custom models - TCG A57 non regressions. Light testing for TCG though. Deep review may detect some mistakes when migrating between named fields and IDRegMap storage - light testing of introspection. Testing a given writable ID field value with query-cpu-model-expansion is not supported yet. TODO/QUESTIONS: - Some idreg named fields are not yet migrated to an array storage. some of them are not in isar struct either. Maybe we could have handled TCG and KVM separately and it may turn out that this conversion is unneeded. So as it is quite cumbersome I prefered to keep it for a later stage. - the custom model does not come with legacy host properties such as SVE, MTE, expecially those that induce some KVM settings. This needs to be fixed. - The custom model and its exposed properties depend on the host capabilities. More and more IDREG become writable meaning that the custom model gains more properties over the time and it is host linux dependent. At the moment there is no versioning in place. By default the custom model is a host passthrough model (besides the legacy functions). So if the end-user tries to set a field that is not writable from a kernel pov, it will fail. Nevertheless a versionned custom model could constrain the props exposed, independently on the host linux capabilities. - the QEMU layer does not take care of IDREG field value consistency. The kernel neither. I imagine this could be the role of the upper layer to implement a vcpu profile that makes sure settings are consistent. Here we come to "named" models. What should they look like on ARM? - Implementation details: - it seems there are a lot of duplications in the code. ID regs are described in different manners, with different data structs, for TCG, now for KVM. - The IdRegMap->regs is sparsely populated. Maybe a better data struct could be used, although this is the one chosen for the kernel uapi. References: [1] [PATCH v12 00/11] Support writable CPU ID registers from userspace https://lore.kernel.org/all/20230609190054.1542113-1-oliver.upton@linux.dev/ [2] qemu-system-aarch64 -qmp unix:/home/augere/TEST/QEMU/qmp-sock,server,nowait -M virt --enable-kvm -cpu custom scripts/qmp/qmp-shell /home/augere/TEST/QEMU/qmp-sock Welcome to the QMP low-level shell! Connected to QEMU 9.0.50 (QEMU) query-cpu-model-expansion type=full model={"name":"custom"} [3] KVM_CAP_ARM_SUPPORTED_REG_MASK_RANGES KVM_ARM_GET_REG_WRITABLE_MASKS Documentation/virt/kvm/api.rst [4] linux "sysreg" file linux/arch/arm64/tools/sysreg and gen-sysreg.awk ./tools/include/generated/asm/sysreg-defs.h Cornelia Huck (4): kvm: kvm_get_writable_id_regs virt: Allow custom vcpu model in arm virt arm-qmp-cmds: introspection for custom model arm/cpu-features: Document custom vcpu model Eric Auger (17): arm/cpu: Add sysreg definitions in cpu-sysegs.h arm/cpu: Store aa64isar0 into the idregs arrays arm/cpu: Store aa64isar1/2 into the idregs array arm/cpu: Store aa64drf0/1 into the idregs array arm/cpu: Store aa64mmfr0-3 into the idregs array arm/cpu: Store aa64drf0/1 into the idregs array arm/cpu: Store aa64smfr0 into the idregs array arm/cpu: Store id_isar0-7 into the idregs array arm/cpu: Store id_mfr0/1 into the idregs array arm/cpu: Store id_dfr0/1 into the idregs array arm/cpu: Store id_mmfr0-5 into the idregs array arm/cpu: Add infra to handle generated ID register definitions arm/cpu: Add sysreg generation scripts arm/cpu: Add generated files arm/kvm: Allow reading all the writable ID registers arm/kvm: write back modified ID regs to KVM arm/cpu: Introduce a customizable kvm host cpu model docs/system/arm/cpu-features.rst | 55 ++- target/arm/cpu-custom.h | 58 +++ target/arm/cpu-features.h | 307 ++++++------ target/arm/cpu-sysregs.h | 152 ++++++ target/arm/cpu.h | 120 +++-- target/arm/internals.h | 6 +- target/arm/kvm_arm.h | 16 +- hw/arm/virt.c | 3 + hw/intc/armv7m_nvic.c | 27 +- target/arm/arm-qmp-cmds.c | 56 ++- target/arm/cpu-sysreg-properties.c | 682 ++++++++++++++++++++++++++ target/arm/cpu.c | 124 +++-- target/arm/cpu64.c | 265 +++++++--- target/arm/helper.c | 68 +-- target/arm/kvm.c | 253 +++++++--- target/arm/ptw.c | 6 +- target/arm/tcg/cpu-v7m.c | 174 +++---- target/arm/tcg/cpu32.c | 320 ++++++------ target/arm/tcg/cpu64.c | 460 ++++++++--------- scripts/gen-cpu-sysreg-properties.awk | 325 ++++++++++++ scripts/gen-cpu-sysregs-header.awk | 47 ++ scripts/update-aarch64-sysreg-code.sh | 27 + target/arm/meson.build | 1 + target/arm/trace-events | 8 + 24 files changed, 2646 insertions(+), 914 deletions(-) create mode 100644 target/arm/cpu-custom.h create mode 100644 target/arm/cpu-sysregs.h create mode 100644 target/arm/cpu-sysreg-properties.c create mode 100755 scripts/gen-cpu-sysreg-properties.awk create mode 100755 scripts/gen-cpu-sysregs-header.awk create mode 100755 scripts/update-aarch64-sysreg-code.sh