[0/9] userfaultfd: add minor fault handling

Message

Axel Rasmussen Jan. 15, 2021, 7:04 p.m. UTC

Changelog
=========

RFC->v1:
- Rebased onto Peter Xu's patches for disabling huge PMD sharing for certain
  userfaultfd-registered areas.
- Added commits which update documentation, and add a self test which exercises
  the new feature.
- Fixed reporting CONTINUE as a supported ioctl even for non-MINOR ranges.

Overview
========

This series adds a new userfaultfd registration mode,
UFFDIO_REGISTER_MODE_MINOR. This allows userspace to intercept "minor" faults.
By "minor" fault, I mean the following situation:

Let there exist two mappings (i.e., VMAs) to the same page(s) (shared memory).
One of the mappings is registered with userfaultfd (in minor mode), and the
other is not. Via the non-UFFD mapping, the underlying pages have already been
allocated & filled with some contents. The UFFD mapping has not yet been
faulted in; when it is touched for the first time, this results in what I'm
calling a "minor" fault. As a concrete example, when working with hugetlbfs, we
have huge_pte_none(), but find_lock_page() finds an existing page.

We also add a new ioctl to resolve such faults: UFFDIO_CONTINUE. The idea is,
userspace resolves the fault by either a) doing nothing if the contents are
already correct, or b) updating the underlying contents using the second,
non-UFFD mapping (via memcpy/memset or similar, or something fancier like RDMA,
or etc...). In either case, userspace issues UFFDIO_CONTINUE to tell the kernel
"I have ensured the page contents are correct, carry on setting up the mapping".

Use Case
========

Consider the use case of VM live migration (e.g. under QEMU/KVM):

1. While a VM is still running, we copy the contents of its memory to a
   target machine. The pages are populated on the target by writing to the
   non-UFFD mapping, using the setup described above. The VM is still running
   (and therefore its memory is likely changing), so this may be repeated
   several times, until we decide the target is "up to date enough".

2. We pause the VM on the source, and start executing on the target machine.
   During this gap, the VM's user(s) will *see* a pause, so it is desirable to
   minimize this window.

3. Between the last time any page was copied from the source to the target, and
   when the VM was paused, the contents of that page may have changed - and
   therefore the copy we have on the target machine is out of date. Although we
   can keep track of which pages are out of date, for VMs with large amounts of
   memory, it is "slow" to transfer this information to the target machine. We
   want to resume execution before such a transfer would complete.

4. So, the guest begins executing on the target machine. The first time it
   touches its memory (via the UFFD-registered mapping), userspace wants to
   intercept this fault. Userspace checks whether or not the page is up to date,
   and if not, copies the updated page from the source machine, via the non-UFFD
   mapping. Finally, whether a copy was performed or not, userspace issues a
   UFFDIO_CONTINUE ioctl to tell the kernel "I have ensured the page contents
   are correct, carry on setting up the mapping".

We don't have to do all of the final updates on-demand. The userfaultfd manager
can, in the background, also copy over updated pages once it receives the map of
which pages are up-to-date or not.

Interaction with Existing APIs
==============================

Because it's possible to combine registration modes (e.g. a single VMA can be
userfaultfd-registered MINOR | MISSING), and because it's up to userspace how to
resolve faults once they are received, I spent some time thinking through how
the existing API interacts with the new feature.

UFFDIO_CONTINUE cannot be used to resolve non-minor faults, as it does not
allocate a new page. If UFFDIO_CONTINUE is used on a non-minor fault:

- For non-shared memory or shmem, -EINVAL is returned.
- For hugetlb, -EFAULT is returned.

UFFDIO_COPY and UFFDIO_ZEROPAGE cannot be used to resolve minor faults. Without
modifications, the existing codepath assumes a new page needs to be allocated.
This is okay, since userspace must have a second non-UFFD-registered mapping
anyway, thus there isn't much reason to want to use these in any case (just
memcpy or memset or similar).

- If UFFDIO_COPY is used on a minor fault, -EEXIST is returned.
- If UFFDIO_ZEROPAGE is used on a minor fault, -EEXIST is returned (or -EINVAL
  in the case of hugetlb, as UFFDIO_ZEROPAGE is unsupported in any case).
- UFFDIO_WRITEPROTECT simply doesn't work with shared memory, and returns
  -ENOENT in that case (regardless of the kind of fault).

Dependencies
============

I've included 4 commits from Peter Xu's larger series
(https://lore.kernel.org/patchwork/cover/1366017/) in this series. My changes
depend on his work, to disable huge PMD sharing for MINOR registered userfaultfd
areas. I included the 4 commits directly because a) it lets this series just be
applied and work as-is, and b) they are fairly standalone, and could potentially
be merged even without the rest of the larger series Peter submitted. Thanks
Peter!

Also, although it doesn't affect minor fault handling, I did notice that the
userfaultfd self test sometimes experienced memory corruption
(https://lore.kernel.org/patchwork/cover/1356755/). For anyone testing this
series, it may be useful to apply that series first to fix the selftest
flakiness. That series doesn't have to be merged into mainline / maintaner
branches before mine, though.

Future Work
===========

Currently the patchset only supports hugetlbfs. There is no reason it can't work
with shmem, but I expect hugetlbfs to be much more commonly used since we're
talking about backing guest memory for VMs. I plan to implement shmem support in
a follow-up patch series.

Axel Rasmussen (5):
  userfaultfd: add minor fault registration mode
  userfaultfd: disable huge PMD sharing for MINOR registered VMAs
  userfaultfd: add UFFDIO_CONTINUE ioctl
  userfaultfd: update documentation to describe minor fault handling
  userfaultfd/selftests: add test exercising minor fault handling

Peter Xu (4):
  hugetlb: Pass vma into huge_pte_alloc()
  hugetlb/userfaultfd: Forbid huge pmd sharing when uffd enabled
  mm/hugetlb: Move flush_hugetlb_tlb_range() into hugetlb.h
  hugetlb/userfaultfd: Unshare all pmds for hugetlbfs when register wp

 Documentation/admin-guide/mm/userfaultfd.rst | 105 ++++++----
 arch/arm64/mm/hugetlbpage.c                  |   5 +-
 arch/ia64/mm/hugetlbpage.c                   |   3 +-
 arch/mips/mm/hugetlbpage.c                   |   4 +-
 arch/parisc/mm/hugetlbpage.c                 |   2 +-
 arch/powerpc/mm/hugetlbpage.c                |   3 +-
 arch/s390/mm/hugetlbpage.c                   |   2 +-
 arch/sh/mm/hugetlbpage.c                     |   2 +-
 arch/sparc/mm/hugetlbpage.c                  |   2 +-
 fs/proc/task_mmu.c                           |   1 +
 fs/userfaultfd.c                             | 190 ++++++++++++++++---
 include/linux/hugetlb.h                      |  22 ++-
 include/linux/mm.h                           |   1 +
 include/linux/mmu_notifier.h                 |   1 +
 include/linux/userfaultfd_k.h                |  29 ++-
 include/trace/events/mmflags.h               |   1 +
 include/uapi/linux/userfaultfd.h             |  36 +++-
 mm/hugetlb.c                                 |  61 ++++--
 mm/userfaultfd.c                             |  88 ++++++---
 tools/testing/selftests/vm/userfaultfd.c     | 147 +++++++++++++-
 20 files changed, 570 insertions(+), 135 deletions(-)

--
2.30.0.284.gd98b1dd5eaa7-goog

Comments

Peter Xu Jan. 21, 2021, 7:12 p.m. UTC | #1

On Fri, Jan 15, 2021 at 11:04:42AM -0800, Axel Rasmussen wrote:
> UFFDIO_COPY and UFFDIO_ZEROPAGE cannot be used to resolve minor faults. Without
> modifications, the existing codepath assumes a new page needs to be allocated.
> This is okay, since userspace must have a second non-UFFD-registered mapping
> anyway, thus there isn't much reason to want to use these in any case (just
> memcpy or memset or similar).
> 
> - If UFFDIO_COPY is used on a minor fault, -EEXIST is returned.

When minor fault the dst VM will report to src with the address.  The src could
checkup whether dst contains the latest data on that (pmd) page and either:

  - it's latest, then tells dst, dst does UFFDIO_CONTINUE

  - it's not latest, then tells dst (probably along with the page data?  if
    hugetlbfs doesn't support double map, we'd need to batch all the dirty
    small pages in one shot), dst does whatever to replace the page

Then, I'm thinking what would be the way to replace an old page.. is that one
FALLOC_FL_PUNCH_HOLE plus one UFFDIO_COPY at last?

Thanks,

Axel Rasmussen Jan. 21, 2021, 10:13 p.m. UTC | #2

On Thu, Jan 21, 2021 at 11:12 AM Peter Xu <[e-mail]> wrote:
>
> On Fri, Jan 15, 2021 at 11:04:42AM -0800, Axel Rasmussen wrote:
> > UFFDIO_COPY and UFFDIO_ZEROPAGE cannot be used to resolve minor faults. Without
> > modifications, the existing codepath assumes a new page needs to be allocated.
> > This is okay, since userspace must have a second non-UFFD-registered mapping
> > anyway, thus there isn't much reason to want to use these in any case (just
> > memcpy or memset or similar).
> >
> > - If UFFDIO_COPY is used on a minor fault, -EEXIST is returned.
>
> When minor fault the dst VM will report to src with the address.  The src could
> checkup whether dst contains the latest data on that (pmd) page and either:
>
>   - it's latest, then tells dst, dst does UFFDIO_CONTINUE
>
>   - it's not latest, then tells dst (probably along with the page data?  if
>     hugetlbfs doesn't support double map, we'd need to batch all the dirty
>     small pages in one shot), dst does whatever to replace the page
>
> Then, I'm thinking what would be the way to replace an old page.. is that one
> FALLOC_FL_PUNCH_HOLE plus one UFFDIO_COPY at last?

When I wrote this, my thinking was that users of this feature would
have two mappings, one of which is not UFFD registered at all. So, to
replace the existing page contents, userspace would just write to the
non-UFFD mapping (with memcpy() or whatever else, or we could get
fancy and imagine using some RDMA technology to copy the page over the
network from the live migration source directly in place). After
performing the write, we just UFFDIO_CONTINUE.

I believe FALLOC_FL_PUNCH_HOLE / MADV_REMOVE doesn't work with
hugetlbfs? Once shmem support is implemented, I would expect
FALLOC_FL_PUNCH_HOLE + UFFDIO_COPY to work, but I wonder if such an
operation would be more expensive than just copying using the other
side of the shared mapping?

>
> Thanks,
>
> --
> Peter Xu
>

Peter Xu Jan. 21, 2021, 10:37 p.m. UTC | #3

On Thu, Jan 21, 2021 at 02:13:50PM -0800, Axel Rasmussen wrote:
> When I wrote this, my thinking was that users of this feature would
> have two mappings, one of which is not UFFD registered at all. So, to
> replace the existing page contents, userspace would just write to the
> non-UFFD mapping (with memcpy() or whatever else, or we could get
> fancy and imagine using some RDMA technology to copy the page over the
> network from the live migration source directly in place). After
> performing the write, we just UFFDIO_CONTINUE.
> 
> I believe FALLOC_FL_PUNCH_HOLE / MADV_REMOVE doesn't work with
> hugetlbfs? Once shmem support is implemented, I would expect
> FALLOC_FL_PUNCH_HOLE + UFFDIO_COPY to work, but I wonder if such an
> operation would be more expensive than just copying using the other
> side of the shared mapping?

IIUC hugetlb supports that (hugetlbfs_punch_hole()).  But I agree with you on
what you said should be good enough.  Thanks,