| Message ID | 20220303193842.370645-1-pbonzini@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | KVM: x86/mmu: Overhaul TDP MMU zapping and flushing | expand |
On 3/3/22 20:38, Paolo Bonzini wrote: > > Overhaul TDP MMU's handling of zapping and TLB flushing to reduce the > number of TLB flushes, fix soft lockups and RCU stalls, avoid blocking > vCPUs for long durations while zapping paging structure, and to clean up > the zapping code. > > The largest cleanup is to separate the flows for zapping roots (zap > _everything_), zapping leaf SPTEs (zap guest mappings for whatever reason), > and zapping a specific SP (NX recovery). They're currently smushed into a > single zap_gfn_range(), which was a good idea at the time, but became a > mess when trying to handle the different rules, e.g. TLB flushes aren't > needed when zapping a root because KVM can safely zap a root if and only > if it's unreachable. > > To solve the soft lockups, stalls, and vCPU performance issues: > > - Defer remote TLB flushes to the caller when zapping TDP MMU shadow > pages by relying on RCU to ensure the paging structure isn't freed > until all vCPUs have exited the guest. > > - Allowing yielding when zapping TDP MMU roots in response to the root's > last reference being put. This requires a bit of trickery to ensure > the root is reachable via mmu_notifier, but it's not too gross. > > - Zap roots in two passes to avoid holding RCU for potential hundreds of > seconds when zapping guest with terabytes of memory that is backed > entirely by 4kb SPTEs. > > - Zap defunct roots asynchronously via the common work_queue so that a > vCPU doesn't get stuck doing the work if the vCPU happens to drop the > last reference to a root. > > The selftest at the end allows populating a guest with the max amount of > memory allowed by the underlying architecture. The most I've tested is > ~64tb (MAXPHYADDR=46) as I don't have easy access to a system with > MAXPHYADDR=52. The selftest compiles on arm64 and s390x, but otherwise > hasn't been tested outside of x86-64. It will hopefully do something > useful as is, but there's a non-zero chance it won't get past init with > a high max memory. Running on x86 without the TDP MMU is comically slow. > > Testing: passes kvm-unit-tests and guest installation tests on Intel. > Haven't yet run AMD or selftests. > > Thanks, > > Paolo > > v4: > - collected reviews and typo fixes (plus some typo fixes of my own) > > - new patches to simplify reader invariants: they are not allowed to > acquire references to invalid roots > > - new version of "Allow yielding when zapping GFNs for defunct TDP MMU > root", simplifying the atomic a bit by 1) using xchg and relying on > its implicit memory barriers 2) relying on readers to have the same > behavior for the three stats refcount=0/valid, refcount=0/invalid, > refcount=1/invalid (see previous point) > > - switch zapping of invalidated roots to asynchronous workers on a > per-VM workqueue, fixing a bug in v3 where the extra reference added > by kvm_tdp_mmu_put_root could be given back twice. This also replaces > "KVM: x86/mmu: Use common iterator for walking invalid TDP MMU roots" > in v3, since it gets rid of next_invalidated_root() in a different way. > > - because of the previous point, most of the logic in v3's "KVM: x86/mmu: > Zap defunct roots via asynchronous worker" moves to the earlier patch > "KVM: x86/mmu: Zap invalidated roots via asynchronous worker" > > > v3: > - Drop patches that were applied. > - Rebase to latest kvm/queue. > - Collect a review. [David] > - Use helper instead of goto to zap roots in two passes. [David] > - Add patches to disallow REMOVED "old" SPTE when atomically > setting SPTE. > > Paolo Bonzini (5): > KVM: x86/mmu: only perform eager page splitting on valid roots > KVM: x86/mmu: do not allow readers to acquire references to invalid roots > KVM: x86/mmu: Zap invalidated roots via asynchronous worker > KVM: x86/mmu: Allow yielding when zapping GFNs for defunct TDP MMU root > KVM: x86/mmu: Zap defunct roots via asynchronous worker > > Sean Christopherson (25): > KVM: x86/mmu: Check for present SPTE when clearing dirty bit in TDP MMU > KVM: x86/mmu: Fix wrong/misleading comments in TDP MMU fast zap > KVM: x86/mmu: Formalize TDP MMU's (unintended?) deferred TLB flush logic > KVM: x86/mmu: Document that zapping invalidated roots doesn't need to flush > KVM: x86/mmu: Require mmu_lock be held for write in unyielding root iter > KVM: x86/mmu: Check for !leaf=>leaf, not PFN change, in TDP MMU SP removal > KVM: x86/mmu: Batch TLB flushes from TDP MMU for MMU notifier change_spte > KVM: x86/mmu: Drop RCU after processing each root in MMU notifier hooks > KVM: x86/mmu: Add helpers to read/write TDP MMU SPTEs and document RCU > KVM: x86/mmu: WARN if old _or_ new SPTE is REMOVED in non-atomic path > KVM: x86/mmu: Refactor low-level TDP MMU set SPTE helper to take raw values > KVM: x86/mmu: Zap only the target TDP MMU shadow page in NX recovery > KVM: x86/mmu: Skip remote TLB flush when zapping all of TDP MMU > KVM: x86/mmu: Add dedicated helper to zap TDP MMU root shadow page > KVM: x86/mmu: Require mmu_lock be held for write to zap TDP MMU range > KVM: x86/mmu: Zap only TDP MMU leafs in kvm_zap_gfn_range() > KVM: x86/mmu: Do remote TLB flush before dropping RCU in TDP MMU resched > KVM: x86/mmu: Defer TLB flush to caller when freeing TDP MMU shadow pages > KVM: x86/mmu: Zap roots in two passes to avoid inducing RCU stalls > KVM: x86/mmu: Check for a REMOVED leaf SPTE before making the SPTE > KVM: x86/mmu: WARN on any attempt to atomically update REMOVED SPTE > KVM: selftests: Move raw KVM_SET_USER_MEMORY_REGION helper to utils > KVM: selftests: Split out helper to allocate guest mem via memfd > KVM: selftests: Define cpu_relax() helpers for s390 and x86 > KVM: selftests: Add test to populate a VM with the max possible guest mem > > arch/x86/include/asm/kvm_host.h | 2 + > arch/x86/kvm/mmu/mmu.c | 49 +- > arch/x86/kvm/mmu/mmu_internal.h | 15 +- > arch/x86/kvm/mmu/tdp_iter.c | 6 +- > arch/x86/kvm/mmu/tdp_iter.h | 15 +- > arch/x86/kvm/mmu/tdp_mmu.c | 559 +++++++++++------- > arch/x86/kvm/mmu/tdp_mmu.h | 26 +- > tools/testing/selftests/kvm/.gitignore | 1 + > tools/testing/selftests/kvm/Makefile | 3 + > .../selftests/kvm/include/kvm_util_base.h | 5 + > .../selftests/kvm/include/s390x/processor.h | 8 + > .../selftests/kvm/include/x86_64/processor.h | 5 + > tools/testing/selftests/kvm/lib/kvm_util.c | 66 ++- > .../selftests/kvm/max_guest_memory_test.c | 292 +++++++++ > .../selftests/kvm/set_memory_region_test.c | 35 +- > 15 files changed, 794 insertions(+), 293 deletions(-) > create mode 100644 tools/testing/selftests/kvm/max_guest_memory_test.c > Queued, thanks. Paolo