[v2,00/12] Implement Eager Page Splitting for ARM.

Message

Ricardo Koller Feb. 6, 2023, 4:58 p.m. UTC

Eager Page Splitting improves the performance of dirty-logging (used
in live migrations) when guest memory is backed by huge-pages.  It's
an optimization used in Google Cloud since 2016 on x86, and for the
last couple of months on ARM.

Background and motivation
=========================
Dirty logging is typically used for live-migration iterative copying.
KVM implements dirty-logging at the PAGE_SIZE granularity (will refer
to 4K pages from now on).  It does it by faulting on write-protected
4K pages.  Therefore, enabling dirty-logging on a huge-page requires
breaking it into 4K pages in the first place.  KVM does this breaking
on fault, and because it's in the critical path it only maps the 4K
page that faulted; every other 4K page is left unmapped.  This is not
great for performance on ARM for a couple of reasons:

- Splitting on fault can halt vcpus for milliseconds in some
  implementations. Splitting a block PTE requires using a broadcasted
  TLB invalidation (TLBI) for every huge-page (due to the
  break-before-make requirement). Note that x86 doesn't need this. We
  observed some implementations that take millliseconds to complete
  broadcasted TLBIs when done in parallel from multiple vcpus.  And
  that's exactly what happens when doing it on fault: multiple vcpus
  fault at the same time triggering TLBIs in parallel.

- Read intensive guest workloads end up paying for dirty-logging.
  Only mapping the faulting 4K page means that all the other pages
  that were part of the huge-page will now be unmapped. The effect is
  that any access, including reads, now has to fault.

Eager Page Splitting (on ARM)
=============================
Eager Page Splitting fixes the above two issues by eagerly splitting
huge-pages when enabling dirty logging. The goal is to avoid doing it
while faulting on write-protected pages. This is what the TDP MMU does
for x86 [0], except that x86 does it for different reasons: to avoid
grabbing the MMU lock on fault. Note that taking care of
write-protection faults still requires grabbing the MMU lock on ARM,
but not on x86 (with the fast_page_fault path).

An additional benefit of eagerly splitting huge-pages is that it can
be done in a controlled way (e.g., via an IOCTL). This series provides
two knobs for doing it, just like its x86 counterpart: when enabling
dirty logging, and when using the KVM_CLEAR_DIRTY_LOG ioctl. The
benefit of doing it on KVM_CLEAR_DIRTY_LOG is that this ioctl takes
ranges, and not complete memslots like when enabling dirty logging.
This means that the cost of splitting (mainly broadcasted TLBIs) can
be throttled: split a range, wait for a bit, split another range, etc.
The benefits of this approach were presented by Oliver Upton at KVM
Forum 2022 [1].

Implementation
==============
Patches 3-4 add a pgtable utility function for splitting huge block
PTEs: kvm_pgtable_stage2_split(). Patches 5-9 add support for eagerly
splitting huge-pages when enabling dirty-logging and when using the
KVM_CLEAR_DIRTY_LOG ioctl. Note that this is just like what x86 does,
and the code is actually based on it.  And finally, patch 9:

	KVM: arm64: Use local TLBI on permission relaxation

adds support for using local TLBIs instead of broadcasts when doing
permission relaxation. This last patch is key to achieving good
performance during dirty-logging, as eagerly breaking huge-pages
replaces mapping new pages with permission relaxation. Got this patch
(indirectly) from Marc Z.  and took the liberty of adding a commit
message.

Note: this applies on top of 6.2-rc6.

Performance evaluation
======================
The performance benefits were tested using the dirty_log_perf_test
selftest with 2M huge-pages.

The first test uses a write-only sequential workload where the stride
is 2M instead of 4K [2]. The idea with this experiment is to emulate a
random access pattern writing a different huge-page at every access.
Observe that the benefit increases with the number of vcpus: up to
5.76x for 152 vcpus. This table shows the guest dirtying time when
using the CLEAR ioctl (and KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2):

/dirty_log_perf_test_sparse -s anonymous_hugetlb_2mb -b 1G -v $i -i 3 -m 2

	+-------+----------+------------------+
	| vCPUs | 6.2-rc3  | 6.2-rc3 + series |
	|       |    (ms)  |             (ms) |
	+-------+----------+------------------+
	|    1  |    2.63  |          1.66    |
	|    2  |    2.95  |          1.70    |
	|    4  |    3.21  |          1.71    |
	|    8  |    4.97  |          1.78    |
	|   16  |    9.51  |          1.82    |
	|   32  |   20.15  |          3.03    |
	|   64  |   40.09  |          5.80    |
	|  128  |   80.08  |         12.24    |
	|  152  |  109.81  |         15.14    |
	+-------+----------+------------------+

This secondv test measures the benefit of eager page splitting on read
intensive workloads (1 write for every 10 reads). As in the other
test, the benefit increases with the number of vcpus, up to 8.82x for
152 vcpus. This table shows the guest dirtying time when using the
CLEAR ioctl (and KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2):

./dirty_log_perf_test -s anonymous_hugetlb_2mb -b 1G -v $i -i 3 -m 2 -w 10

	+-------+----------+------------------+
	| vCPUs | 6.2-rc3  | 6.2-rc3 + series |
	|       |   (sec)  |            (sec) |
	+-------+----------+------------------+
	|    1  |    0.65  |          0.07    |
	|    2  |    0.70  |          0.08    |
	|    4  |    0.71  |          0.08    |
	|    8  |    0.72  |          0.08    |
	|   16  |    0.76  |          0.08    |
	|   32  |    1.61  |          0.14    |
	|   64  |    3.46  |          0.30    |
	|  128  |    5.49  |          0.64    |
	|  152  |    6.44  |          0.63    |
	+-------+----------+------------------+

Changes from v1:
https://lore.kernel.org/kvmarm/[e-mail]/
- added a capability to set the eager splitting chunk size. This
  indirectly sets the number of pages in the cache. It also allows for
  opting out of this feature. (Oliver, Marc)
- changed kvm_pgtable_stage2_split() to split 1g huge-pages
  using either 513 or 1 at a time (with a cache of 1). (Oliver, Marc)
- added force_pte arg to kvm_pgtable_stage2_create_removed().
- renamed free_removed to free_unlinked. (Ben and Oliver)
- added KVM_PGTABLE_WALK ctx->flags for skipping BBM and CMO, instead
  of KVM_PGTABLE_WALK_REMOVED. (Oliver)

Changes from the RFC:
https://lore.kernel.org/kvmarm/[e-mail]/
- dropped the changes to split on POST visits. No visible perf
  benefit.
- changed the kvm_pgtable_stage2_free_removed() implementation to
  reuse the stage2 mapper.
- dropped the FEAT_BBM changes and optimization. Will send this on a
  different series.

Thanks,
Ricardo

[0] https://lore.kernel.org/kvm/[e-mail]/
[1] https://kvmforum2022.sched.com/event/15jJq/kvmarm-at-scale-improvements-to-the-mmu-in-the-face-of-hardware-growing-pains-oliver-upton-google
[2] https://github.com/ricarkol/linux/commit/f78e9102b2bff4fb7f30bee810d7d611a537b46d
[3] https://lore.kernel.org/kvmarm/[e-mail]/

Marc Zyngier (1):
  KVM: arm64: Use local TLBI on permission relaxation

Ricardo Koller (11):
  KVM: arm64: Add KVM_PGTABLE_WALK ctx->flags for skipping BBM and CMO
  KVM: arm64: Rename free_unlinked to free_removed
  KVM: arm64: Add helper for creating unlinked stage2 subtrees
  KVM: arm64: Add kvm_pgtable_stage2_split()
  KVM: arm64: Refactor kvm_arch_commit_memory_region()
  KVM: arm64: Add kvm_uninit_stage2_mmu()
  KVM: arm64: Export kvm_are_all_memslots_empty()
  KVM: arm64: Add KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE
  KVM: arm64: Split huge pages when dirty logging is enabled
  KVM: arm64: Open-code kvm_mmu_write_protect_pt_masked()
  KVM: arm64: Split huge pages during KVM_CLEAR_DIRTY_LOG

 Documentation/virt/kvm/api.rst        |  26 ++++
 arch/arm64/include/asm/kvm_asm.h      |   4 +
 arch/arm64/include/asm/kvm_host.h     |  18 +++
 arch/arm64/include/asm/kvm_mmu.h      |   1 +
 arch/arm64/include/asm/kvm_pgtable.h  |  85 +++++++++++-
 arch/arm64/kvm/arm.c                  |  22 +++
 arch/arm64/kvm/hyp/nvhe/hyp-main.c    |  10 ++
 arch/arm64/kvm/hyp/nvhe/mem_protect.c |   6 +-
 arch/arm64/kvm/hyp/nvhe/tlb.c         |  54 ++++++++
 arch/arm64/kvm/hyp/pgtable.c          | 187 +++++++++++++++++++++++---
 arch/arm64/kvm/hyp/vhe/tlb.c          |  32 +++++
 arch/arm64/kvm/mmu.c                  | 183 ++++++++++++++++++++-----
 include/linux/kvm_host.h              |   2 +
 include/uapi/linux/kvm.h              |   1 +
 virt/kvm/kvm_main.c                   |   2 +-
 15 files changed, 579 insertions(+), 54 deletions(-)

Comments

Gavin Shan Feb. 14, 2023, 5:57 a.m. UTC | #1

Hi Ricardo,

On 2/7/23 3:58 AM, Ricardo Koller wrote:
> Eager Page Splitting improves the performance of dirty-logging (used
> in live migrations) when guest memory is backed by huge-pages.  It's
> an optimization used in Google Cloud since 2016 on x86, and for the
> last couple of months on ARM.
> 
> Background and motivation
> =========================
> Dirty logging is typically used for live-migration iterative copying.
> KVM implements dirty-logging at the PAGE_SIZE granularity (will refer
> to 4K pages from now on).  It does it by faulting on write-protected
> 4K pages.  Therefore, enabling dirty-logging on a huge-page requires
> breaking it into 4K pages in the first place.  KVM does this breaking
> on fault, and because it's in the critical path it only maps the 4K
> page that faulted; every other 4K page is left unmapped.  This is not
> great for performance on ARM for a couple of reasons:
> 
> - Splitting on fault can halt vcpus for milliseconds in some
>    implementations. Splitting a block PTE requires using a broadcasted
>    TLB invalidation (TLBI) for every huge-page (due to the
>    break-before-make requirement). Note that x86 doesn't need this. We
>    observed some implementations that take millliseconds to complete
>    broadcasted TLBIs when done in parallel from multiple vcpus.  And
>    that's exactly what happens when doing it on fault: multiple vcpus
>    fault at the same time triggering TLBIs in parallel.
> 
> - Read intensive guest workloads end up paying for dirty-logging.
>    Only mapping the faulting 4K page means that all the other pages
>    that were part of the huge-page will now be unmapped. The effect is
>    that any access, including reads, now has to fault.
> 
> Eager Page Splitting (on ARM)
> =============================
> Eager Page Splitting fixes the above two issues by eagerly splitting
> huge-pages when enabling dirty logging. The goal is to avoid doing it
> while faulting on write-protected pages. This is what the TDP MMU does
> for x86 [0], except that x86 does it for different reasons: to avoid
> grabbing the MMU lock on fault. Note that taking care of
> write-protection faults still requires grabbing the MMU lock on ARM,
> but not on x86 (with the fast_page_fault path).
> 
> An additional benefit of eagerly splitting huge-pages is that it can
> be done in a controlled way (e.g., via an IOCTL). This series provides
> two knobs for doing it, just like its x86 counterpart: when enabling
> dirty logging, and when using the KVM_CLEAR_DIRTY_LOG ioctl. The
> benefit of doing it on KVM_CLEAR_DIRTY_LOG is that this ioctl takes
> ranges, and not complete memslots like when enabling dirty logging.
> This means that the cost of splitting (mainly broadcasted TLBIs) can
> be throttled: split a range, wait for a bit, split another range, etc.
> The benefits of this approach were presented by Oliver Upton at KVM
> Forum 2022 [1].
> 

[...]

Sorry for raising questions about the design lately. There are two operations
regarding the existing huge page mapping. Here, lets take PMD and PTE mapping
as an example for discussion: (a) The existing PMD mapping is split to contiguous
512 PTE mappings when all sub-pages are written in sequence and dirty logging has
been enabled (b) The contiguous 512 PTE mappings are combined to one PMD mapping
when dirty logging is disabled.

Before this series is applied, both (a) and (b) are handled by the page fault handler.
After this series is applied, (a) is handled in the ioctl handler while (b) is still
handled in the page fault handler. I'm not sure why we can't eagerly split the PMD
mapping into 512 PTE mapping in the page fault handler? In this way, the implementation
may be simplified by extending kvm_pgtable_stage2_map(). In the implementation, the
newly introduced API kvm_pgtable_stage2_split() calls to kvm_pgtable_stage2_create_unlinked()
and then stage2_map_walker(), which is part of kvm_pgtable_stage2_map(), to create the
unlinked page tables. It's why I have the question.

Thanks,
Gavin

Oliver Upton Feb. 14, 2023, 7:33 a.m. UTC | #2

Hi Gavin,

On Tue, Feb 14, 2023 at 04:57:59PM +1100, Gavin Shan wrote:
> On 2/7/23 3:58 AM, Ricardo Koller wrote:

<snip>

> > Eager Page Splitting fixes the above two issues by eagerly splitting
> > huge-pages when enabling dirty logging. The goal is to avoid doing it
> > while faulting on write-protected pages.

</snip>

> I'm not sure why we can't eagerly split the PMD mapping into 512 PTE
> mapping in the page fault handler?

The entire goal of the series is to avoid page splitting at all on the
stage-2 abort path. Ideally we want to minimize the time taken to handle
a fault so we can get back to running the guest. The requirement to
perform a break-before-make operation to change the mapping granularity
can, as Ricardo points out, be a bottleneck on contemporary implementations.

There is a clear uplift with the proposed implementation already, and I
would expect that margin to widen if/when we add support for lockless
(i.e. RCU-protected) permission relaxation.

> In the implementation, the newly introduced API
> kvm_pgtable_stage2_split() calls to kvm_pgtable_stage2_create_unlinked()
> and then stage2_map_walker(), which is part of kvm_pgtable_stage2_map(),
> to create the unlinked page tables.

This is deliberate code reuse. Page table construction in the fault path
is largely similar to that of eager split besides the fact that one is
working on 'live' page tables whereas the other is not. As such I gave
the suggestion to Ricardo to reuse what we have today for the sake of
eager splitting.

Ricardo Koller Feb. 14, 2023, 9:59 p.m. UTC | #3

On Mon, Feb 13, 2023 at 11:33 PM Oliver Upton <[e-mail]> wrote:
>
> Hi Gavin,
>
> On Tue, Feb 14, 2023 at 04:57:59PM +1100, Gavin Shan wrote:
> > On 2/7/23 3:58 AM, Ricardo Koller wrote:
>
> <snip>
>
> > > Eager Page Splitting fixes the above two issues by eagerly splitting
> > > huge-pages when enabling dirty logging. The goal is to avoid doing it
> > > while faulting on write-protected pages.
>
> </snip>
>
> > I'm not sure why we can't eagerly split the PMD mapping into 512 PTE
> > mapping in the page fault handler?
>
> The entire goal of the series is to avoid page splitting at all on the
> stage-2 abort path. Ideally we want to minimize the time taken to handle
> a fault so we can get back to running the guest. The requirement to
> perform a break-before-make operation to change the mapping granularity
> can, as Ricardo points out, be a bottleneck on contemporary implementations.
>
> There is a clear uplift with the proposed implementation already, and I
> would expect that margin to widen if/when we add support for lockless
> (i.e. RCU-protected) permission relaxation.

There's also the issue of allocating 513 pages on fault when splitting PUDs.

>
> > In the implementation, the newly introduced API
> > kvm_pgtable_stage2_split() calls to kvm_pgtable_stage2_create_unlinked()
> > and then stage2_map_walker(), which is part of kvm_pgtable_stage2_map(),
> > to create the unlinked page tables.
>
> This is deliberate code reuse. Page table construction in the fault path
> is largely similar to that of eager split besides the fact that one is
> working on 'live' page tables whereas the other is not. As such I gave
> the suggestion to Ricardo to reuse what we have today for the sake of
> eager splitting.
>
> --
> Thanks,
> Oliver