[00/16] KVM: arm64: nv: Shadow stage-2 page table handling

Message

Marc Zyngier April 9, 2024, 5:54 p.m. UTC

Here's another instalment of everyone's favourite "arm64 nested virt,
one headache at a time". This time, we deal with the shadowing of the
guest's S2 page tables.

So here's the 10000m (approximately 30000ft for those of you stuck
with the wrong units) view of what this is doing:

- for each {VMID,VTTBR,VTCR} tuple the guest uses, we use a separate
  shadow s2_mmu context. This context has its own "real" VMID and a
  set of page tables that are the combination of the guest's S2 and
  the host S2, built dynamically one fault at a time.

- these shadow S2 contexts are ephemeral, and behave exactly as
  TLBs. For all intent and purposes, they *are* TLBs, and we discard
  them pretty often.

- TLB invalidation takes three possible paths:

  * either this is an EL2 S1 invalidation, and we directly emulate it
    as early as possible

  * or this is an EL1 S1 invalidation, and we need to apply it to the
    shadow S2s (plural!) that match the VMID set by the L1 guest

  * or finally, this is affecting S2, and we need to teardown the
    corresponding part of the shadow S2s, which invalidates the TLBs

From a quality of implementation, this series does the absolute
minimum. In a lot of cases, we blow away all the shadow S2s without
any discrimination. That's because we don't have a reverse mapping
yet, so if something gets unmapped from the canonical S2 through a MMU
notifier, things slow down significantly. At this stage, nobody should
care.

We also make some implementation choices:

- no overhead for non-NV guests -- this is our #1 requirement

- all the TLBIs are implemented as Inner-Shareable, no matter what the
  guest says

- we don't try to optimise for leaf invalidation at S2

- we use a TTL-like mechanism to limit the over-invalidation when no
  TTL is provided, but this is only a best effort process

- range invalidation is supported

- NXS operations are supported as well, and implemented as XS. Nobody
  cares about them anyway

Note that some of the patches used to carry review tags, but the
series has had so many changes that they are not making sense anymore.
This is based on 6.9-rc3, and has been tested on my usual M2 with the
rest of the NV series.

Christoffer Dall (2):
  KVM: arm64: nv: Implement nested Stage-2 page table walk logic
  KVM: arm64: nv: Unmap/flush shadow stage 2 page tables

Marc Zyngier (14):
  KVM: arm64: nv: Support multiple nested Stage-2 mmu structures
  KVM: arm64: nv: Handle shadow stage 2 page faults
  KVM: arm64: nv: Add Stage-1 EL2 invalidation primitives
  KVM: arm64: nv: Handle EL2 Stage-1 TLB invalidation
  KVM: arm64: nv: Handle TLB invalidation targeting L2 stage-1
  KVM: arm64: nv: Handle TLBI VMALLS12E1{,IS} operations
  KVM: arm64: nv: Handle TLBI ALLE1{,IS} operations
  KVM: arm64: nv: Handle TLBI IPAS2E1{,IS} operations
  KVM: arm64: nv: Handle FEAT_TTL hinted TLB operations
  KVM: arm64: nv: Tag shadow S2 entries with guest's leaf S2 level
  KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like
    information
  KVM: arm64: nv: Add handling of outer-shareable TLBI operations
  KVM: arm64: nv: Add handling of range-based TLBI operations
  KVM: arm64: nv: Add handling of NXS-flavoured TLBI operations

 arch/arm64/include/asm/esr.h         |   1 +
 arch/arm64/include/asm/kvm_asm.h     |   2 +
 arch/arm64/include/asm/kvm_emulate.h |   1 +
 arch/arm64/include/asm/kvm_host.h    |  41 ++
 arch/arm64/include/asm/kvm_mmu.h     |  12 +
 arch/arm64/include/asm/kvm_nested.h  | 127 +++++
 arch/arm64/include/asm/sysreg.h      |  17 +
 arch/arm64/kvm/arm.c                 |  11 +
 arch/arm64/kvm/hyp/vhe/switch.c      |  51 +-
 arch/arm64/kvm/hyp/vhe/tlb.c         | 147 +++++
 arch/arm64/kvm/mmu.c                 | 219 ++++++--
 arch/arm64/kvm/nested.c              | 767 ++++++++++++++++++++++++++-
 arch/arm64/kvm/reset.c               |   6 +
 arch/arm64/kvm/sys_regs.c            | 398 ++++++++++++++
 14 files changed, 1759 insertions(+), 41 deletions(-)