| Message ID | 20231009064230.2952396-3-surenb@google.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | userfaultfd move option | expand |
On 09.10.23 08:42, Suren Baghdasaryan wrote: > From: Andrea Arcangeli <aarcange@redhat.com> > > Implement the uABI of UFFDIO_MOVE ioctl. > UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application > needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are > available (in userspace) for recycling, as is usually the case in heap > compaction algorithms, then we can avoid the page allocation and memcpy > (done by UFFDIO_COPY). Also, since the pages are recycled in the > userspace, we avoid the need to release (via madvise) the pages back to > the kernel [2]. > We see over 40% reduction (on a Google pixel 6 device) in the compacting > thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was > measured using a benchmark that emulates a heap compaction implementation > using userfaultfd (to allow concurrent accesses by application threads). > More details of the usecase are explained in [2]. > Furthermore, UFFDIO_MOVE enables moving swapped-out pages without > touching them within the same vma. Today, it can only be done by mremap, > however it forces splitting the vma. > > [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ > [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ > > Update for the ioctl_userfaultfd(2) manpage: > > UFFDIO_MOVE > (Since Linux xxx) Move a continuous memory chunk into the > userfault registered range and optionally wake up the blocked > thread. The source and destination addresses and the number of > bytes to move are specified by the src, dst, and len fields of > the uffdio_move structure pointed to by argp: > > struct uffdio_move { > __u64 dst; /* Destination of move */ > __u64 src; /* Source of move */ > __u64 len; /* Number of bytes to move */ > __u64 mode; /* Flags controlling behavior of move */ > __s64 move; /* Number of bytes moved, or negated error */ > }; > > The following value may be bitwise ORed in mode to change the > behavior of the UFFDIO_MOVE operation: > > UFFDIO_MOVE_MODE_DONTWAKE > Do not wake up the thread that waits for page-fault > resolution > > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES > Allow holes in the source virtual range that is being moved. > When not specified, the holes will result in ENOENT error. > When specified, the holes will be accounted as successfully > moved memory. This is mostly useful to move hugepage aligned > virtual regions without knowing if there are transparent > hugepages in the regions or not, but preventing the risk of > having to split the hugepage during the operation. > > The move field is used by the kernel to return the number of > bytes that was actually moved, or an error (a negated errno- > style value). If the value returned in move doesn't match the > value that was specified in len, the operation fails with the > error EAGAIN. The move field is output-only; it is not read by > the UFFDIO_MOVE operation. > > The operation may fail for various reasons. Usually, remapping of > pages that are not exclusive to the given process fail; once KSM > might deduplicate pages or fork() COW-shares pages during fork() > with child processes, they are no longer exclusive. Further, the > kernel might only perform lightweight checks for detecting whether > the pages are exclusive, and return -EBUSY in case that check fails. > To make the operation more likely to succeed, KSM should be > disabled, fork() should be avoided or MADV_DONTFORK should be > configured for the source VMA before fork(). > > This ioctl(2) operation returns 0 on success. In this case, the > entire area was moved. On error, -1 is returned and errno is > set to indicate the error. Possible errors include: > > EAGAIN The number of bytes moved (i.e., the value returned in > the move field) does not equal the value that was > specified in the len field. > > EINVAL Either dst or len was not a multiple of the system page > size, or the range specified by src and len or dst and len > was invalid. > > EINVAL An invalid bit was specified in the mode field. > > ENOENT > The source virtual memory range has unmapped holes and > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. > > EEXIST > The destination virtual memory range is fully or partially > mapped. > > EBUSY > The pages in the source virtual memory range are not > exclusive to the process. The kernel might only perform > lightweight checks for detecting whether the pages are > exclusive. To make the operation more likely to succeed, > KSM should be disabled, fork() should be avoided or > MADV_DONTFORK should be configured for the source virtual > memory area before fork(). > > ENOMEM Allocating memory needed for the operation failed. > > ESRCH > The faulting process has exited at the time of a > UFFDIO_MOVE operation. > A general comment simply because I realized that just now: does anything speak against limiting the operations now to a single MM? The use cases I heard so far don't need it. If ever required, we could consider extending it. Let's reduce complexity and KIS unless really required. Further: see "22) Do not crash the kernel" in coding-style.rst. All these BUG_ON need to go. Ideally, use WARN_ON_ONCE() or just VM_WARN_ON().
On Mon, Oct 9, 2023 at 7:38 AM David Hildenbrand <david@redhat.com> wrote: > > On 09.10.23 08:42, Suren Baghdasaryan wrote: > > From: Andrea Arcangeli <aarcange@redhat.com> > > > > Implement the uABI of UFFDIO_MOVE ioctl. > > UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application > > needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are > > available (in userspace) for recycling, as is usually the case in heap > > compaction algorithms, then we can avoid the page allocation and memcpy > > (done by UFFDIO_COPY). Also, since the pages are recycled in the > > userspace, we avoid the need to release (via madvise) the pages back to > > the kernel [2]. > > We see over 40% reduction (on a Google pixel 6 device) in the compacting > > thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was > > measured using a benchmark that emulates a heap compaction implementation > > using userfaultfd (to allow concurrent accesses by application threads). > > More details of the usecase are explained in [2]. > > Furthermore, UFFDIO_MOVE enables moving swapped-out pages without > > touching them within the same vma. Today, it can only be done by mremap, > > however it forces splitting the vma. > > > > [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ > > [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ > > > > Update for the ioctl_userfaultfd(2) manpage: > > > > UFFDIO_MOVE > > (Since Linux xxx) Move a continuous memory chunk into the > > userfault registered range and optionally wake up the blocked > > thread. The source and destination addresses and the number of > > bytes to move are specified by the src, dst, and len fields of > > the uffdio_move structure pointed to by argp: > > > > struct uffdio_move { > > __u64 dst; /* Destination of move */ > > __u64 src; /* Source of move */ > > __u64 len; /* Number of bytes to move */ > > __u64 mode; /* Flags controlling behavior of move */ > > __s64 move; /* Number of bytes moved, or negated error */ > > }; > > > > The following value may be bitwise ORed in mode to change the > > behavior of the UFFDIO_MOVE operation: > > > > UFFDIO_MOVE_MODE_DONTWAKE > > Do not wake up the thread that waits for page-fault > > resolution > > > > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES > > Allow holes in the source virtual range that is being moved. > > When not specified, the holes will result in ENOENT error. > > When specified, the holes will be accounted as successfully > > moved memory. This is mostly useful to move hugepage aligned > > virtual regions without knowing if there are transparent > > hugepages in the regions or not, but preventing the risk of > > having to split the hugepage during the operation. > > > > The move field is used by the kernel to return the number of > > bytes that was actually moved, or an error (a negated errno- > > style value). If the value returned in move doesn't match the > > value that was specified in len, the operation fails with the > > error EAGAIN. The move field is output-only; it is not read by > > the UFFDIO_MOVE operation. > > > > The operation may fail for various reasons. Usually, remapping of > > pages that are not exclusive to the given process fail; once KSM > > might deduplicate pages or fork() COW-shares pages during fork() > > with child processes, they are no longer exclusive. Further, the > > kernel might only perform lightweight checks for detecting whether > > the pages are exclusive, and return -EBUSY in case that check fails. > > To make the operation more likely to succeed, KSM should be > > disabled, fork() should be avoided or MADV_DONTFORK should be > > configured for the source VMA before fork(). > > > > This ioctl(2) operation returns 0 on success. In this case, the > > entire area was moved. On error, -1 is returned and errno is > > set to indicate the error. Possible errors include: > > > > EAGAIN The number of bytes moved (i.e., the value returned in > > the move field) does not equal the value that was > > specified in the len field. > > > > EINVAL Either dst or len was not a multiple of the system page > > size, or the range specified by src and len or dst and len > > was invalid. > > > > EINVAL An invalid bit was specified in the mode field. > > > > ENOENT > > The source virtual memory range has unmapped holes and > > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. > > > > EEXIST > > The destination virtual memory range is fully or partially > > mapped. > > > > EBUSY > > The pages in the source virtual memory range are not > > exclusive to the process. The kernel might only perform > > lightweight checks for detecting whether the pages are > > exclusive. To make the operation more likely to succeed, > > KSM should be disabled, fork() should be avoided or > > MADV_DONTFORK should be configured for the source virtual > > memory area before fork(). > > > > ENOMEM Allocating memory needed for the operation failed. > > > > ESRCH > > The faulting process has exited at the time of a > > UFFDIO_MOVE operation. > > > > A general comment simply because I realized that just now: does anything > speak against limiting the operations now to a single MM? > > The use cases I heard so far don't need it. If ever required, we could > consider extending it. > > Let's reduce complexity and KIS unless really required. Let me check if there are use cases that require moves between MMs. Andrea seems to have put considerable effort to make it work between MMs and it would be a pity to lose that. I can send a follow-up patch to recover that functionality and even if it does not get merged, it can be used in the future as a reference. But first let me check if we can drop it. > > > Further: see "22) Do not crash the kernel" in coding-style.rst. All > these BUG_ON need to go. Ideally, use WARN_ON_ONCE() or just VM_WARN_ON(). Yeah, it might be the right time to clean that up. Will do. Thanks, Suren. > > -- > Cheers, > > David / dhildenb >
On 09.10.23 18:21, Suren Baghdasaryan wrote: > On Mon, Oct 9, 2023 at 7:38 AM David Hildenbrand <david@redhat.com> wrote: >> >> On 09.10.23 08:42, Suren Baghdasaryan wrote: >>> From: Andrea Arcangeli <aarcange@redhat.com> >>> >>> Implement the uABI of UFFDIO_MOVE ioctl. >>> UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application >>> needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are >>> available (in userspace) for recycling, as is usually the case in heap >>> compaction algorithms, then we can avoid the page allocation and memcpy >>> (done by UFFDIO_COPY). Also, since the pages are recycled in the >>> userspace, we avoid the need to release (via madvise) the pages back to >>> the kernel [2]. >>> We see over 40% reduction (on a Google pixel 6 device) in the compacting >>> thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was >>> measured using a benchmark that emulates a heap compaction implementation >>> using userfaultfd (to allow concurrent accesses by application threads). >>> More details of the usecase are explained in [2]. >>> Furthermore, UFFDIO_MOVE enables moving swapped-out pages without >>> touching them within the same vma. Today, it can only be done by mremap, >>> however it forces splitting the vma. >>> >>> [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ >>> [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ >>> >>> Update for the ioctl_userfaultfd(2) manpage: >>> >>> UFFDIO_MOVE >>> (Since Linux xxx) Move a continuous memory chunk into the >>> userfault registered range and optionally wake up the blocked >>> thread. The source and destination addresses and the number of >>> bytes to move are specified by the src, dst, and len fields of >>> the uffdio_move structure pointed to by argp: >>> >>> struct uffdio_move { >>> __u64 dst; /* Destination of move */ >>> __u64 src; /* Source of move */ >>> __u64 len; /* Number of bytes to move */ >>> __u64 mode; /* Flags controlling behavior of move */ >>> __s64 move; /* Number of bytes moved, or negated error */ >>> }; >>> >>> The following value may be bitwise ORed in mode to change the >>> behavior of the UFFDIO_MOVE operation: >>> >>> UFFDIO_MOVE_MODE_DONTWAKE >>> Do not wake up the thread that waits for page-fault >>> resolution >>> >>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES >>> Allow holes in the source virtual range that is being moved. >>> When not specified, the holes will result in ENOENT error. >>> When specified, the holes will be accounted as successfully >>> moved memory. This is mostly useful to move hugepage aligned >>> virtual regions without knowing if there are transparent >>> hugepages in the regions or not, but preventing the risk of >>> having to split the hugepage during the operation. >>> >>> The move field is used by the kernel to return the number of >>> bytes that was actually moved, or an error (a negated errno- >>> style value). If the value returned in move doesn't match the >>> value that was specified in len, the operation fails with the >>> error EAGAIN. The move field is output-only; it is not read by >>> the UFFDIO_MOVE operation. >>> >>> The operation may fail for various reasons. Usually, remapping of >>> pages that are not exclusive to the given process fail; once KSM >>> might deduplicate pages or fork() COW-shares pages during fork() >>> with child processes, they are no longer exclusive. Further, the >>> kernel might only perform lightweight checks for detecting whether >>> the pages are exclusive, and return -EBUSY in case that check fails. >>> To make the operation more likely to succeed, KSM should be >>> disabled, fork() should be avoided or MADV_DONTFORK should be >>> configured for the source VMA before fork(). >>> >>> This ioctl(2) operation returns 0 on success. In this case, the >>> entire area was moved. On error, -1 is returned and errno is >>> set to indicate the error. Possible errors include: >>> >>> EAGAIN The number of bytes moved (i.e., the value returned in >>> the move field) does not equal the value that was >>> specified in the len field. >>> >>> EINVAL Either dst or len was not a multiple of the system page >>> size, or the range specified by src and len or dst and len >>> was invalid. >>> >>> EINVAL An invalid bit was specified in the mode field. >>> >>> ENOENT >>> The source virtual memory range has unmapped holes and >>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. >>> >>> EEXIST >>> The destination virtual memory range is fully or partially >>> mapped. >>> >>> EBUSY >>> The pages in the source virtual memory range are not >>> exclusive to the process. The kernel might only perform >>> lightweight checks for detecting whether the pages are >>> exclusive. To make the operation more likely to succeed, >>> KSM should be disabled, fork() should be avoided or >>> MADV_DONTFORK should be configured for the source virtual >>> memory area before fork(). >>> >>> ENOMEM Allocating memory needed for the operation failed. >>> >>> ESRCH >>> The faulting process has exited at the time of a >>> UFFDIO_MOVE operation. >>> >> >> A general comment simply because I realized that just now: does anything >> speak against limiting the operations now to a single MM? >> >> The use cases I heard so far don't need it. If ever required, we could >> consider extending it. >> >> Let's reduce complexity and KIS unless really required. > > Let me check if there are use cases that require moves between MMs. > Andrea seems to have put considerable effort to make it work between > MMs and it would be a pity to lose that. I can send a follow-up patch > to recover that functionality and even if it does not get merged, it > can be used in the future as a reference. But first let me check if we > can drop it. Yes, that sounds reasonable. Unless the big important use cases requires moving pages between processes, let's leave that as future work for now.
On Mon, Oct 9, 2023 at 5:24 PM David Hildenbrand <david@redhat.com> wrote: > > On 09.10.23 18:21, Suren Baghdasaryan wrote: > > On Mon, Oct 9, 2023 at 7:38 AM David Hildenbrand <david@redhat.com> wrote: > >> > >> On 09.10.23 08:42, Suren Baghdasaryan wrote: > >>> From: Andrea Arcangeli <aarcange@redhat.com> > >>> > >>> Implement the uABI of UFFDIO_MOVE ioctl. > >>> UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application > >>> needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are > >>> available (in userspace) for recycling, as is usually the case in heap > >>> compaction algorithms, then we can avoid the page allocation and memcpy > >>> (done by UFFDIO_COPY). Also, since the pages are recycled in the > >>> userspace, we avoid the need to release (via madvise) the pages back to > >>> the kernel [2]. > >>> We see over 40% reduction (on a Google pixel 6 device) in the compacting > >>> thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was > >>> measured using a benchmark that emulates a heap compaction implementation > >>> using userfaultfd (to allow concurrent accesses by application threads). > >>> More details of the usecase are explained in [2]. > >>> Furthermore, UFFDIO_MOVE enables moving swapped-out pages without > >>> touching them within the same vma. Today, it can only be done by mremap, > >>> however it forces splitting the vma. > >>> > >>> [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ > >>> [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ > >>> > >>> Update for the ioctl_userfaultfd(2) manpage: > >>> > >>> UFFDIO_MOVE > >>> (Since Linux xxx) Move a continuous memory chunk into the > >>> userfault registered range and optionally wake up the blocked > >>> thread. The source and destination addresses and the number of > >>> bytes to move are specified by the src, dst, and len fields of > >>> the uffdio_move structure pointed to by argp: > >>> > >>> struct uffdio_move { > >>> __u64 dst; /* Destination of move */ > >>> __u64 src; /* Source of move */ > >>> __u64 len; /* Number of bytes to move */ > >>> __u64 mode; /* Flags controlling behavior of move */ > >>> __s64 move; /* Number of bytes moved, or negated error */ > >>> }; > >>> > >>> The following value may be bitwise ORed in mode to change the > >>> behavior of the UFFDIO_MOVE operation: > >>> > >>> UFFDIO_MOVE_MODE_DONTWAKE > >>> Do not wake up the thread that waits for page-fault > >>> resolution > >>> > >>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES > >>> Allow holes in the source virtual range that is being moved. > >>> When not specified, the holes will result in ENOENT error. > >>> When specified, the holes will be accounted as successfully > >>> moved memory. This is mostly useful to move hugepage aligned > >>> virtual regions without knowing if there are transparent > >>> hugepages in the regions or not, but preventing the risk of > >>> having to split the hugepage during the operation. > >>> > >>> The move field is used by the kernel to return the number of > >>> bytes that was actually moved, or an error (a negated errno- > >>> style value). If the value returned in move doesn't match the > >>> value that was specified in len, the operation fails with the > >>> error EAGAIN. The move field is output-only; it is not read by > >>> the UFFDIO_MOVE operation. > >>> > >>> The operation may fail for various reasons. Usually, remapping of > >>> pages that are not exclusive to the given process fail; once KSM > >>> might deduplicate pages or fork() COW-shares pages during fork() > >>> with child processes, they are no longer exclusive. Further, the > >>> kernel might only perform lightweight checks for detecting whether > >>> the pages are exclusive, and return -EBUSY in case that check fails. > >>> To make the operation more likely to succeed, KSM should be > >>> disabled, fork() should be avoided or MADV_DONTFORK should be > >>> configured for the source VMA before fork(). > >>> > >>> This ioctl(2) operation returns 0 on success. In this case, the > >>> entire area was moved. On error, -1 is returned and errno is > >>> set to indicate the error. Possible errors include: > >>> > >>> EAGAIN The number of bytes moved (i.e., the value returned in > >>> the move field) does not equal the value that was > >>> specified in the len field. > >>> > >>> EINVAL Either dst or len was not a multiple of the system page > >>> size, or the range specified by src and len or dst and len > >>> was invalid. > >>> > >>> EINVAL An invalid bit was specified in the mode field. > >>> > >>> ENOENT > >>> The source virtual memory range has unmapped holes and > >>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. > >>> > >>> EEXIST > >>> The destination virtual memory range is fully or partially > >>> mapped. > >>> > >>> EBUSY > >>> The pages in the source virtual memory range are not > >>> exclusive to the process. The kernel might only perform > >>> lightweight checks for detecting whether the pages are > >>> exclusive. To make the operation more likely to succeed, > >>> KSM should be disabled, fork() should be avoided or > >>> MADV_DONTFORK should be configured for the source virtual > >>> memory area before fork(). > >>> > >>> ENOMEM Allocating memory needed for the operation failed. > >>> > >>> ESRCH > >>> The faulting process has exited at the time of a > >>> UFFDIO_MOVE operation. > >>> > >> > >> A general comment simply because I realized that just now: does anything > >> speak against limiting the operations now to a single MM? > >> > >> The use cases I heard so far don't need it. If ever required, we could > >> consider extending it. > >> > >> Let's reduce complexity and KIS unless really required. > > > > Let me check if there are use cases that require moves between MMs. > > Andrea seems to have put considerable effort to make it work between > > MMs and it would be a pity to lose that. I can send a follow-up patch > > to recover that functionality and even if it does not get merged, it > > can be used in the future as a reference. But first let me check if we > > can drop it. For the compaction use case that we have it's fine to limit it to single MM. However, for general use I think Peter will have a better idea. > > Yes, that sounds reasonable. Unless the big important use cases requires > moving pages between processes, let's leave that as future work for now. > > -- > Cheers, > > David / dhildenb >
On Mon, Oct 9, 2023 at 9:29 AM Lokesh Gidra <lokeshgidra@google.com> wrote: > > On Mon, Oct 9, 2023 at 5:24 PM David Hildenbrand <david@redhat.com> wrote: > > > > On 09.10.23 18:21, Suren Baghdasaryan wrote: > > > On Mon, Oct 9, 2023 at 7:38 AM David Hildenbrand <david@redhat.com> wrote: > > >> > > >> On 09.10.23 08:42, Suren Baghdasaryan wrote: > > >>> From: Andrea Arcangeli <aarcange@redhat.com> > > >>> > > >>> Implement the uABI of UFFDIO_MOVE ioctl. > > >>> UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application > > >>> needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are > > >>> available (in userspace) for recycling, as is usually the case in heap > > >>> compaction algorithms, then we can avoid the page allocation and memcpy > > >>> (done by UFFDIO_COPY). Also, since the pages are recycled in the > > >>> userspace, we avoid the need to release (via madvise) the pages back to > > >>> the kernel [2]. > > >>> We see over 40% reduction (on a Google pixel 6 device) in the compacting > > >>> thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was > > >>> measured using a benchmark that emulates a heap compaction implementation > > >>> using userfaultfd (to allow concurrent accesses by application threads). > > >>> More details of the usecase are explained in [2]. > > >>> Furthermore, UFFDIO_MOVE enables moving swapped-out pages without > > >>> touching them within the same vma. Today, it can only be done by mremap, > > >>> however it forces splitting the vma. > > >>> > > >>> [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ > > >>> [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ > > >>> > > >>> Update for the ioctl_userfaultfd(2) manpage: > > >>> > > >>> UFFDIO_MOVE > > >>> (Since Linux xxx) Move a continuous memory chunk into the > > >>> userfault registered range and optionally wake up the blocked > > >>> thread. The source and destination addresses and the number of > > >>> bytes to move are specified by the src, dst, and len fields of > > >>> the uffdio_move structure pointed to by argp: > > >>> > > >>> struct uffdio_move { > > >>> __u64 dst; /* Destination of move */ > > >>> __u64 src; /* Source of move */ > > >>> __u64 len; /* Number of bytes to move */ > > >>> __u64 mode; /* Flags controlling behavior of move */ > > >>> __s64 move; /* Number of bytes moved, or negated error */ > > >>> }; > > >>> > > >>> The following value may be bitwise ORed in mode to change the > > >>> behavior of the UFFDIO_MOVE operation: > > >>> > > >>> UFFDIO_MOVE_MODE_DONTWAKE > > >>> Do not wake up the thread that waits for page-fault > > >>> resolution > > >>> > > >>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES > > >>> Allow holes in the source virtual range that is being moved. > > >>> When not specified, the holes will result in ENOENT error. > > >>> When specified, the holes will be accounted as successfully > > >>> moved memory. This is mostly useful to move hugepage aligned > > >>> virtual regions without knowing if there are transparent > > >>> hugepages in the regions or not, but preventing the risk of > > >>> having to split the hugepage during the operation. > > >>> > > >>> The move field is used by the kernel to return the number of > > >>> bytes that was actually moved, or an error (a negated errno- > > >>> style value). If the value returned in move doesn't match the > > >>> value that was specified in len, the operation fails with the > > >>> error EAGAIN. The move field is output-only; it is not read by > > >>> the UFFDIO_MOVE operation. > > >>> > > >>> The operation may fail for various reasons. Usually, remapping of > > >>> pages that are not exclusive to the given process fail; once KSM > > >>> might deduplicate pages or fork() COW-shares pages during fork() > > >>> with child processes, they are no longer exclusive. Further, the > > >>> kernel might only perform lightweight checks for detecting whether > > >>> the pages are exclusive, and return -EBUSY in case that check fails. > > >>> To make the operation more likely to succeed, KSM should be > > >>> disabled, fork() should be avoided or MADV_DONTFORK should be > > >>> configured for the source VMA before fork(). > > >>> > > >>> This ioctl(2) operation returns 0 on success. In this case, the > > >>> entire area was moved. On error, -1 is returned and errno is > > >>> set to indicate the error. Possible errors include: > > >>> > > >>> EAGAIN The number of bytes moved (i.e., the value returned in > > >>> the move field) does not equal the value that was > > >>> specified in the len field. > > >>> > > >>> EINVAL Either dst or len was not a multiple of the system page > > >>> size, or the range specified by src and len or dst and len > > >>> was invalid. > > >>> > > >>> EINVAL An invalid bit was specified in the mode field. > > >>> > > >>> ENOENT > > >>> The source virtual memory range has unmapped holes and > > >>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. > > >>> > > >>> EEXIST > > >>> The destination virtual memory range is fully or partially > > >>> mapped. > > >>> > > >>> EBUSY > > >>> The pages in the source virtual memory range are not > > >>> exclusive to the process. The kernel might only perform > > >>> lightweight checks for detecting whether the pages are > > >>> exclusive. To make the operation more likely to succeed, > > >>> KSM should be disabled, fork() should be avoided or > > >>> MADV_DONTFORK should be configured for the source virtual > > >>> memory area before fork(). > > >>> > > >>> ENOMEM Allocating memory needed for the operation failed. > > >>> > > >>> ESRCH > > >>> The faulting process has exited at the time of a > > >>> UFFDIO_MOVE operation. > > >>> > > >> > > >> A general comment simply because I realized that just now: does anything > > >> speak against limiting the operations now to a single MM? > > >> > > >> The use cases I heard so far don't need it. If ever required, we could > > >> consider extending it. > > >> > > >> Let's reduce complexity and KIS unless really required. > > > > > > Let me check if there are use cases that require moves between MMs. > > > Andrea seems to have put considerable effort to make it work between > > > MMs and it would be a pity to lose that. I can send a follow-up patch > > > to recover that functionality and even if it does not get merged, it > > > can be used in the future as a reference. But first let me check if we > > > can drop it. > > For the compaction use case that we have it's fine to limit it to > single MM. However, for general use I think Peter will have a better > idea. > > > > Yes, that sounds reasonable. Unless the big important use cases requires > > moving pages between processes, let's leave that as future work for now. > > > > -- > > Cheers, > > > > David / dhildenb > > While going through mremap's move_page_tables code, which is pretty similar to what we do here, I noticed that cache is flushed as well, whereas we are not doing that here. Is that OK? I'm not a MM expert by any means, so it's a question rather than a comment :)
On Mon, Oct 9, 2023 at 5:57 PM Lokesh Gidra <lokeshgidra@google.com> wrote: > > On Mon, Oct 9, 2023 at 9:29 AM Lokesh Gidra <lokeshgidra@google.com> wrote: > > > > On Mon, Oct 9, 2023 at 5:24 PM David Hildenbrand <david@redhat.com> wrote: > > > > > > On 09.10.23 18:21, Suren Baghdasaryan wrote: > > > > On Mon, Oct 9, 2023 at 7:38 AM David Hildenbrand <david@redhat.com> wrote: > > > >> > > > >> On 09.10.23 08:42, Suren Baghdasaryan wrote: > > > >>> From: Andrea Arcangeli <aarcange@redhat.com> > > > >>> > > > >>> Implement the uABI of UFFDIO_MOVE ioctl. > > > >>> UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application > > > >>> needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are > > > >>> available (in userspace) for recycling, as is usually the case in heap > > > >>> compaction algorithms, then we can avoid the page allocation and memcpy > > > >>> (done by UFFDIO_COPY). Also, since the pages are recycled in the > > > >>> userspace, we avoid the need to release (via madvise) the pages back to > > > >>> the kernel [2]. > > > >>> We see over 40% reduction (on a Google pixel 6 device) in the compacting > > > >>> thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was > > > >>> measured using a benchmark that emulates a heap compaction implementation > > > >>> using userfaultfd (to allow concurrent accesses by application threads). > > > >>> More details of the usecase are explained in [2]. > > > >>> Furthermore, UFFDIO_MOVE enables moving swapped-out pages without > > > >>> touching them within the same vma. Today, it can only be done by mremap, > > > >>> however it forces splitting the vma. > > > >>> > > > >>> [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ > > > >>> [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ > > > >>> > > > >>> Update for the ioctl_userfaultfd(2) manpage: > > > >>> > > > >>> UFFDIO_MOVE > > > >>> (Since Linux xxx) Move a continuous memory chunk into the > > > >>> userfault registered range and optionally wake up the blocked > > > >>> thread. The source and destination addresses and the number of > > > >>> bytes to move are specified by the src, dst, and len fields of > > > >>> the uffdio_move structure pointed to by argp: > > > >>> > > > >>> struct uffdio_move { > > > >>> __u64 dst; /* Destination of move */ > > > >>> __u64 src; /* Source of move */ > > > >>> __u64 len; /* Number of bytes to move */ > > > >>> __u64 mode; /* Flags controlling behavior of move */ > > > >>> __s64 move; /* Number of bytes moved, or negated error */ > > > >>> }; > > > >>> > > > >>> The following value may be bitwise ORed in mode to change the > > > >>> behavior of the UFFDIO_MOVE operation: > > > >>> > > > >>> UFFDIO_MOVE_MODE_DONTWAKE > > > >>> Do not wake up the thread that waits for page-fault > > > >>> resolution > > > >>> > > > >>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES > > > >>> Allow holes in the source virtual range that is being moved. > > > >>> When not specified, the holes will result in ENOENT error. > > > >>> When specified, the holes will be accounted as successfully > > > >>> moved memory. This is mostly useful to move hugepage aligned > > > >>> virtual regions without knowing if there are transparent > > > >>> hugepages in the regions or not, but preventing the risk of > > > >>> having to split the hugepage during the operation. > > > >>> > > > >>> The move field is used by the kernel to return the number of > > > >>> bytes that was actually moved, or an error (a negated errno- > > > >>> style value). If the value returned in move doesn't match the > > > >>> value that was specified in len, the operation fails with the > > > >>> error EAGAIN. The move field is output-only; it is not read by > > > >>> the UFFDIO_MOVE operation. > > > >>> > > > >>> The operation may fail for various reasons. Usually, remapping of > > > >>> pages that are not exclusive to the given process fail; once KSM > > > >>> might deduplicate pages or fork() COW-shares pages during fork() > > > >>> with child processes, they are no longer exclusive. Further, the > > > >>> kernel might only perform lightweight checks for detecting whether > > > >>> the pages are exclusive, and return -EBUSY in case that check fails. > > > >>> To make the operation more likely to succeed, KSM should be > > > >>> disabled, fork() should be avoided or MADV_DONTFORK should be > > > >>> configured for the source VMA before fork(). > > > >>> > > > >>> This ioctl(2) operation returns 0 on success. In this case, the > > > >>> entire area was moved. On error, -1 is returned and errno is > > > >>> set to indicate the error. Possible errors include: > > > >>> > > > >>> EAGAIN The number of bytes moved (i.e., the value returned in > > > >>> the move field) does not equal the value that was > > > >>> specified in the len field. > > > >>> > > > >>> EINVAL Either dst or len was not a multiple of the system page > > > >>> size, or the range specified by src and len or dst and len > > > >>> was invalid. > > > >>> > > > >>> EINVAL An invalid bit was specified in the mode field. > > > >>> > > > >>> ENOENT > > > >>> The source virtual memory range has unmapped holes and > > > >>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. > > > >>> > > > >>> EEXIST > > > >>> The destination virtual memory range is fully or partially > > > >>> mapped. > > > >>> > > > >>> EBUSY > > > >>> The pages in the source virtual memory range are not > > > >>> exclusive to the process. The kernel might only perform > > > >>> lightweight checks for detecting whether the pages are > > > >>> exclusive. To make the operation more likely to succeed, > > > >>> KSM should be disabled, fork() should be avoided or > > > >>> MADV_DONTFORK should be configured for the source virtual > > > >>> memory area before fork(). > > > >>> > > > >>> ENOMEM Allocating memory needed for the operation failed. > > > >>> > > > >>> ESRCH > > > >>> The faulting process has exited at the time of a > > > >>> UFFDIO_MOVE operation. > > > >>> > > > >> > > > >> A general comment simply because I realized that just now: does anything > > > >> speak against limiting the operations now to a single MM? > > > >> > > > >> The use cases I heard so far don't need it. If ever required, we could > > > >> consider extending it. > > > >> > > > >> Let's reduce complexity and KIS unless really required. > > > > > > > > Let me check if there are use cases that require moves between MMs. > > > > Andrea seems to have put considerable effort to make it work between > > > > MMs and it would be a pity to lose that. I can send a follow-up patch > > > > to recover that functionality and even if it does not get merged, it > > > > can be used in the future as a reference. But first let me check if we > > > > can drop it. > > > > For the compaction use case that we have it's fine to limit it to > > single MM. However, for general use I think Peter will have a better > > idea. > > > > > > Yes, that sounds reasonable. Unless the big important use cases requires > > > moving pages between processes, let's leave that as future work for now. > > > > > > -- > > > Cheers, > > > > > > David / dhildenb > > > > > While going through mremap's move_page_tables code, which is pretty > similar to what we do here, I noticed that cache is flushed as well, > whereas we are not doing that here. Is that OK? I'm not a MM expert by > any means, so it's a question rather than a comment :) Good question. I'll have to look closer into it. Unfortunately I'll be travelling starting tomorrow and be back next week. Will try my best to answer questions in a timely manner but depends on my connection and availability. Thanks!
On Mon, Oct 09, 2023 at 05:29:08PM +0100, Lokesh Gidra wrote: > On Mon, Oct 9, 2023 at 5:24 PM David Hildenbrand <david@redhat.com> wrote: > > > > On 09.10.23 18:21, Suren Baghdasaryan wrote: > > > On Mon, Oct 9, 2023 at 7:38 AM David Hildenbrand <david@redhat.com> wrote: > > >> > > >> On 09.10.23 08:42, Suren Baghdasaryan wrote: > > >>> From: Andrea Arcangeli <aarcange@redhat.com> > > >>> > > >>> Implement the uABI of UFFDIO_MOVE ioctl. > > >>> UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application > > >>> needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are > > >>> available (in userspace) for recycling, as is usually the case in heap > > >>> compaction algorithms, then we can avoid the page allocation and memcpy > > >>> (done by UFFDIO_COPY). Also, since the pages are recycled in the > > >>> userspace, we avoid the need to release (via madvise) the pages back to > > >>> the kernel [2]. > > >>> We see over 40% reduction (on a Google pixel 6 device) in the compacting > > >>> thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was > > >>> measured using a benchmark that emulates a heap compaction implementation > > >>> using userfaultfd (to allow concurrent accesses by application threads). > > >>> More details of the usecase are explained in [2]. > > >>> Furthermore, UFFDIO_MOVE enables moving swapped-out pages without > > >>> touching them within the same vma. Today, it can only be done by mremap, > > >>> however it forces splitting the vma. > > >>> > > >>> [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ > > >>> [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ > > >>> > > >>> Update for the ioctl_userfaultfd(2) manpage: > > >>> > > >>> UFFDIO_MOVE > > >>> (Since Linux xxx) Move a continuous memory chunk into the > > >>> userfault registered range and optionally wake up the blocked > > >>> thread. The source and destination addresses and the number of > > >>> bytes to move are specified by the src, dst, and len fields of > > >>> the uffdio_move structure pointed to by argp: > > >>> > > >>> struct uffdio_move { > > >>> __u64 dst; /* Destination of move */ > > >>> __u64 src; /* Source of move */ > > >>> __u64 len; /* Number of bytes to move */ > > >>> __u64 mode; /* Flags controlling behavior of move */ > > >>> __s64 move; /* Number of bytes moved, or negated error */ > > >>> }; > > >>> > > >>> The following value may be bitwise ORed in mode to change the > > >>> behavior of the UFFDIO_MOVE operation: > > >>> > > >>> UFFDIO_MOVE_MODE_DONTWAKE > > >>> Do not wake up the thread that waits for page-fault > > >>> resolution > > >>> > > >>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES > > >>> Allow holes in the source virtual range that is being moved. > > >>> When not specified, the holes will result in ENOENT error. > > >>> When specified, the holes will be accounted as successfully > > >>> moved memory. This is mostly useful to move hugepage aligned > > >>> virtual regions without knowing if there are transparent > > >>> hugepages in the regions or not, but preventing the risk of > > >>> having to split the hugepage during the operation. > > >>> > > >>> The move field is used by the kernel to return the number of > > >>> bytes that was actually moved, or an error (a negated errno- > > >>> style value). If the value returned in move doesn't match the > > >>> value that was specified in len, the operation fails with the > > >>> error EAGAIN. The move field is output-only; it is not read by > > >>> the UFFDIO_MOVE operation. > > >>> > > >>> The operation may fail for various reasons. Usually, remapping of > > >>> pages that are not exclusive to the given process fail; once KSM > > >>> might deduplicate pages or fork() COW-shares pages during fork() > > >>> with child processes, they are no longer exclusive. Further, the > > >>> kernel might only perform lightweight checks for detecting whether > > >>> the pages are exclusive, and return -EBUSY in case that check fails. > > >>> To make the operation more likely to succeed, KSM should be > > >>> disabled, fork() should be avoided or MADV_DONTFORK should be > > >>> configured for the source VMA before fork(). > > >>> > > >>> This ioctl(2) operation returns 0 on success. In this case, the > > >>> entire area was moved. On error, -1 is returned and errno is > > >>> set to indicate the error. Possible errors include: > > >>> > > >>> EAGAIN The number of bytes moved (i.e., the value returned in > > >>> the move field) does not equal the value that was > > >>> specified in the len field. > > >>> > > >>> EINVAL Either dst or len was not a multiple of the system page > > >>> size, or the range specified by src and len or dst and len > > >>> was invalid. > > >>> > > >>> EINVAL An invalid bit was specified in the mode field. > > >>> > > >>> ENOENT > > >>> The source virtual memory range has unmapped holes and > > >>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. > > >>> > > >>> EEXIST > > >>> The destination virtual memory range is fully or partially > > >>> mapped. > > >>> > > >>> EBUSY > > >>> The pages in the source virtual memory range are not > > >>> exclusive to the process. The kernel might only perform > > >>> lightweight checks for detecting whether the pages are > > >>> exclusive. To make the operation more likely to succeed, > > >>> KSM should be disabled, fork() should be avoided or > > >>> MADV_DONTFORK should be configured for the source virtual > > >>> memory area before fork(). > > >>> > > >>> ENOMEM Allocating memory needed for the operation failed. > > >>> > > >>> ESRCH > > >>> The faulting process has exited at the time of a > > >>> UFFDIO_MOVE operation. > > >>> > > >> > > >> A general comment simply because I realized that just now: does anything > > >> speak against limiting the operations now to a single MM? > > >> > > >> The use cases I heard so far don't need it. If ever required, we could > > >> consider extending it. > > >> > > >> Let's reduce complexity and KIS unless really required. > > > > > > Let me check if there are use cases that require moves between MMs. > > > Andrea seems to have put considerable effort to make it work between > > > MMs and it would be a pity to lose that. I can send a follow-up patch > > > to recover that functionality and even if it does not get merged, it > > > can be used in the future as a reference. But first let me check if we > > > can drop it. > > For the compaction use case that we have it's fine to limit it to > single MM. However, for general use I think Peter will have a better > idea. I used to have the same thought with David on whether we can simplify the design to e.g. limit it to single mm. Then I found that the trickiest is actually patch 1 together with the anon_vma manipulations, and the problem is that's not avoidable even if we restrict the api to apply on single mm. What else we can benefit from single mm? One less mmap read lock, but probably that's all we can get; IIUC we need to keep most of the rest of the code, e.g. pgtable walks, double pgtable lockings, etc. Actually, even though I have no solid clue, but I had a feeling that there can be some interesting way to leverage this across-mm movement, while keeping things all safe (by e.g. elaborately requiring other proc to create uffd and deliver to this proc). Considering Andrea's original version already contains those bits and all above, I'd vote that we go ahead with supporting two MMs. Thanks,
On Sun, Oct 08, 2023 at 11:42:27PM -0700, Suren Baghdasaryan wrote: > From: Andrea Arcangeli <aarcange@redhat.com> > > Implement the uABI of UFFDIO_MOVE ioctl. > UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application > needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are > available (in userspace) for recycling, as is usually the case in heap > compaction algorithms, then we can avoid the page allocation and memcpy > (done by UFFDIO_COPY). Also, since the pages are recycled in the > userspace, we avoid the need to release (via madvise) the pages back to > the kernel [2]. > We see over 40% reduction (on a Google pixel 6 device) in the compacting > thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was > measured using a benchmark that emulates a heap compaction implementation > using userfaultfd (to allow concurrent accesses by application threads). > More details of the usecase are explained in [2]. > Furthermore, UFFDIO_MOVE enables moving swapped-out pages without > touching them within the same vma. Today, it can only be done by mremap, > however it forces splitting the vma. > > [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ > [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ > > Update for the ioctl_userfaultfd(2) manpage: > > UFFDIO_MOVE > (Since Linux xxx) Move a continuous memory chunk into the > userfault registered range and optionally wake up the blocked > thread. The source and destination addresses and the number of > bytes to move are specified by the src, dst, and len fields of > the uffdio_move structure pointed to by argp: > > struct uffdio_move { > __u64 dst; /* Destination of move */ > __u64 src; /* Source of move */ > __u64 len; /* Number of bytes to move */ > __u64 mode; /* Flags controlling behavior of move */ > __s64 move; /* Number of bytes moved, or negated error */ > }; > > The following value may be bitwise ORed in mode to change the > behavior of the UFFDIO_MOVE operation: > > UFFDIO_MOVE_MODE_DONTWAKE > Do not wake up the thread that waits for page-fault > resolution > > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES > Allow holes in the source virtual range that is being moved. > When not specified, the holes will result in ENOENT error. > When specified, the holes will be accounted as successfully > moved memory. This is mostly useful to move hugepage aligned > virtual regions without knowing if there are transparent > hugepages in the regions or not, but preventing the risk of > having to split the hugepage during the operation. > > The move field is used by the kernel to return the number of > bytes that was actually moved, or an error (a negated errno- > style value). If the value returned in move doesn't match the > value that was specified in len, the operation fails with the > error EAGAIN. The move field is output-only; it is not read by > the UFFDIO_MOVE operation. > > The operation may fail for various reasons. Usually, remapping of > pages that are not exclusive to the given process fail; once KSM > might deduplicate pages or fork() COW-shares pages during fork() > with child processes, they are no longer exclusive. Further, the > kernel might only perform lightweight checks for detecting whether > the pages are exclusive, and return -EBUSY in case that check fails. > To make the operation more likely to succeed, KSM should be > disabled, fork() should be avoided or MADV_DONTFORK should be > configured for the source VMA before fork(). > > This ioctl(2) operation returns 0 on success. In this case, the > entire area was moved. On error, -1 is returned and errno is > set to indicate the error. Possible errors include: > > EAGAIN The number of bytes moved (i.e., the value returned in > the move field) does not equal the value that was > specified in the len field. > > EINVAL Either dst or len was not a multiple of the system page > size, or the range specified by src and len or dst and len > was invalid. > > EINVAL An invalid bit was specified in the mode field. > > ENOENT > The source virtual memory range has unmapped holes and > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. > > EEXIST > The destination virtual memory range is fully or partially > mapped. > > EBUSY > The pages in the source virtual memory range are not > exclusive to the process. The kernel might only perform > lightweight checks for detecting whether the pages are > exclusive. To make the operation more likely to succeed, > KSM should be disabled, fork() should be avoided or > MADV_DONTFORK should be configured for the source virtual > memory area before fork(). > > ENOMEM Allocating memory needed for the operation failed. > > ESRCH > The faulting process has exited at the time of a Nit pick comment for future man page: there's no faulting process in this context. Perhaps "target process"? > UFFDIO_MOVE operation. > > Signed-off-by: Andrea Arcangeli <aarcange@redhat.com> > Signed-off-by: Suren Baghdasaryan <surenb@google.com> > --- > Documentation/admin-guide/mm/userfaultfd.rst | 3 + > fs/userfaultfd.c | 63 ++ > include/linux/rmap.h | 5 + > include/linux/userfaultfd_k.h | 12 + > include/uapi/linux/userfaultfd.h | 29 +- > mm/huge_memory.c | 138 +++++ > mm/khugepaged.c | 3 + > mm/rmap.c | 6 + > mm/userfaultfd.c | 602 +++++++++++++++++++ > 9 files changed, 860 insertions(+), 1 deletion(-) > > diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst > index 203e26da5f92..e5cc8848dcb3 100644 > --- a/Documentation/admin-guide/mm/userfaultfd.rst > +++ b/Documentation/admin-guide/mm/userfaultfd.rst > @@ -113,6 +113,9 @@ events, except page fault notifications, may be generated: > areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the analogous feature indicating > support for shmem virtual memory areas. > > +- ``UFFD_FEATURE_MOVE`` indicates that the kernel supports moving an > + existing page contents from userspace. > + > The userland application should set the feature flags it intends to use > when invoking the ``UFFDIO_API`` ioctl, to request that those features be > enabled if supported. > diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c > index a7c6ef764e63..ac52e0f99a69 100644 > --- a/fs/userfaultfd.c > +++ b/fs/userfaultfd.c > @@ -2039,6 +2039,66 @@ static inline unsigned int uffd_ctx_features(__u64 user_features) > return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED; > } > > +static int userfaultfd_remap(struct userfaultfd_ctx *ctx, > + unsigned long arg) If we do want to rename from REMAP to MOVE, we'd better rename the functions too, as "remap" still exists all over the place.. > +{ > + __s64 ret; > + struct uffdio_move uffdio_move; > + struct uffdio_move __user *user_uffdio_move; > + struct userfaultfd_wake_range range; > + > + user_uffdio_move = (struct uffdio_move __user *) arg; > + > + ret = -EAGAIN; > + if (atomic_read(&ctx->mmap_changing)) > + goto out; I didn't notice this before, but I think we need to re-check this after taking target mm's mmap read lock.. maybe we'd want to pass in ctx* into remap_pages(), more reasoning below. > + > + ret = -EFAULT; > + if (copy_from_user(&uffdio_move, user_uffdio_move, > + /* don't copy "remap" last field */ s/remap/move/ > + sizeof(uffdio_move)-sizeof(__s64))) > + goto out; > + > + ret = validate_range(ctx->mm, uffdio_move.dst, uffdio_move.len); > + if (ret) > + goto out; > + > + ret = validate_range(current->mm, uffdio_move.src, uffdio_move.len); > + if (ret) > + goto out; > + > + ret = -EINVAL; > + if (uffdio_move.mode & ~(UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES| > + UFFDIO_MOVE_MODE_DONTWAKE)) > + goto out; > + > + if (mmget_not_zero(ctx->mm)) { > + ret = remap_pages(ctx->mm, current->mm, > + uffdio_move.dst, uffdio_move.src, > + uffdio_move.len, uffdio_move.mode); > + mmput(ctx->mm); > + } else { > + return -ESRCH; > + } > + > + if (unlikely(put_user(ret, &user_uffdio_move->move))) > + return -EFAULT; > + if (ret < 0) > + goto out; > + > + /* len == 0 would wake all */ > + BUG_ON(!ret); > + range.len = ret; > + if (!(uffdio_move.mode & UFFDIO_MOVE_MODE_DONTWAKE)) { > + range.start = uffdio_move.dst; > + wake_userfault(ctx, &range); > + } > + ret = range.len == uffdio_move.len ? 0 : -EAGAIN; > + > +out: > + return ret; > +} > + > /* > * userland asks for a certain API version and we return which bits > * and ioctl commands are implemented in this kernel for such API > @@ -2131,6 +2191,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd, > case UFFDIO_ZEROPAGE: > ret = userfaultfd_zeropage(ctx, arg); > break; > + case UFFDIO_MOVE: > + ret = userfaultfd_remap(ctx, arg); > + break; > case UFFDIO_WRITEPROTECT: > ret = userfaultfd_writeprotect(ctx, arg); > break; > diff --git a/include/linux/rmap.h b/include/linux/rmap.h > index b26fe858fd44..8034eda972e5 100644 > --- a/include/linux/rmap.h > +++ b/include/linux/rmap.h > @@ -121,6 +121,11 @@ static inline void anon_vma_lock_write(struct anon_vma *anon_vma) > down_write(&anon_vma->root->rwsem); > } > > +static inline int anon_vma_trylock_write(struct anon_vma *anon_vma) > +{ > + return down_write_trylock(&anon_vma->root->rwsem); > +} > + > static inline void anon_vma_unlock_write(struct anon_vma *anon_vma) > { > up_write(&anon_vma->root->rwsem); > diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h > index f2dc19f40d05..ce8d20b57e8c 100644 > --- a/include/linux/userfaultfd_k.h > +++ b/include/linux/userfaultfd_k.h > @@ -93,6 +93,18 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm, > extern long uffd_wp_range(struct vm_area_struct *vma, > unsigned long start, unsigned long len, bool enable_wp); > > +/* remap_pages */ > +void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2); > +void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2); > +ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm, > + unsigned long dst_start, unsigned long src_start, > + unsigned long len, __u64 flags); > +int remap_pages_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, > + pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, > + struct vm_area_struct *dst_vma, > + struct vm_area_struct *src_vma, > + unsigned long dst_addr, unsigned long src_addr); > + > /* mm helpers */ > static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma, > struct vm_userfaultfd_ctx vm_ctx) > diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h > index 0dbc81015018..2841e4ea8f2c 100644 > --- a/include/uapi/linux/userfaultfd.h > +++ b/include/uapi/linux/userfaultfd.h > @@ -41,7 +41,8 @@ > UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \ > UFFD_FEATURE_WP_UNPOPULATED | \ > UFFD_FEATURE_POISON | \ > - UFFD_FEATURE_WP_ASYNC) > + UFFD_FEATURE_WP_ASYNC | \ > + UFFD_FEATURE_MOVE) > #define UFFD_API_IOCTLS \ > ((__u64)1 << _UFFDIO_REGISTER | \ > (__u64)1 << _UFFDIO_UNREGISTER | \ > @@ -50,6 +51,7 @@ > ((__u64)1 << _UFFDIO_WAKE | \ > (__u64)1 << _UFFDIO_COPY | \ > (__u64)1 << _UFFDIO_ZEROPAGE | \ > + (__u64)1 << _UFFDIO_MOVE | \ > (__u64)1 << _UFFDIO_WRITEPROTECT | \ > (__u64)1 << _UFFDIO_CONTINUE | \ > (__u64)1 << _UFFDIO_POISON) > @@ -73,6 +75,7 @@ > #define _UFFDIO_WAKE (0x02) > #define _UFFDIO_COPY (0x03) > #define _UFFDIO_ZEROPAGE (0x04) > +#define _UFFDIO_MOVE (0x05) > #define _UFFDIO_WRITEPROTECT (0x06) > #define _UFFDIO_CONTINUE (0x07) > #define _UFFDIO_POISON (0x08) > @@ -92,6 +95,8 @@ > struct uffdio_copy) > #define UFFDIO_ZEROPAGE _IOWR(UFFDIO, _UFFDIO_ZEROPAGE, \ > struct uffdio_zeropage) > +#define UFFDIO_MOVE _IOWR(UFFDIO, _UFFDIO_MOVE, \ > + struct uffdio_move) > #define UFFDIO_WRITEPROTECT _IOWR(UFFDIO, _UFFDIO_WRITEPROTECT, \ > struct uffdio_writeprotect) > #define UFFDIO_CONTINUE _IOWR(UFFDIO, _UFFDIO_CONTINUE, \ > @@ -222,6 +227,9 @@ struct uffdio_api { > * asynchronous mode is supported in which the write fault is > * automatically resolved and write-protection is un-set. > * It implies UFFD_FEATURE_WP_UNPOPULATED. > + * > + * UFFD_FEATURE_MOVE indicates that the kernel supports moving an > + * existing page contents from userspace. > */ > #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) > #define UFFD_FEATURE_EVENT_FORK (1<<1) > @@ -239,6 +247,7 @@ struct uffdio_api { > #define UFFD_FEATURE_WP_UNPOPULATED (1<<13) > #define UFFD_FEATURE_POISON (1<<14) > #define UFFD_FEATURE_WP_ASYNC (1<<15) > +#define UFFD_FEATURE_MOVE (1<<16) > __u64 features; > > __u64 ioctls; > @@ -347,6 +356,24 @@ struct uffdio_poison { > __s64 updated; > }; > > +struct uffdio_move { > + __u64 dst; > + __u64 src; > + __u64 len; > + /* > + * Especially if used to atomically remove memory from the > + * address space the wake on the dst range is not needed. > + */ > +#define UFFDIO_MOVE_MODE_DONTWAKE ((__u64)1<<0) > +#define UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES ((__u64)1<<1) > + __u64 mode; > + /* > + * "move" is written by the ioctl and must be at the end: the > + * copy_from_user will not read the last 8 bytes. > + */ > + __s64 move; > +}; > + > /* > * Flags for the userfaultfd(2) system call itself. > */ > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > index 9656be95a542..6fac5c3d66e6 100644 > --- a/mm/huge_memory.c > +++ b/mm/huge_memory.c > @@ -2086,6 +2086,144 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, > return ret; > } > > +#ifdef CONFIG_USERFAULTFD > +/* > + * The PT lock for src_pmd and the mmap_lock for reading are held by > + * the caller, but it must return after releasing the > + * page_table_lock. Just move the page from src_pmd to dst_pmd if possible. > + * Return zero if succeeded in moving the page, -EAGAIN if it needs to be > + * repeated by the caller, or other errors in case of failure. > + */ > +int remap_pages_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, > + pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, > + struct vm_area_struct *dst_vma, > + struct vm_area_struct *src_vma, > + unsigned long dst_addr, unsigned long src_addr) > +{ > + pmd_t _dst_pmd, src_pmdval; > + struct page *src_page; > + struct folio *src_folio; > + struct anon_vma *src_anon_vma; > + spinlock_t *src_ptl, *dst_ptl; > + pgtable_t src_pgtable, dst_pgtable; > + struct mmu_notifier_range range; > + int err = 0; > + > + src_pmdval = *src_pmd; > + src_ptl = pmd_lockptr(src_mm, src_pmd); > + > + lockdep_assert_held(src_ptl); > + mmap_assert_locked(src_mm); > + mmap_assert_locked(dst_mm); > + > + BUG_ON(!pmd_none(dst_pmdval)); > + BUG_ON(src_addr & ~HPAGE_PMD_MASK); > + BUG_ON(dst_addr & ~HPAGE_PMD_MASK); > + > + if (!pmd_trans_huge(src_pmdval)) { > + spin_unlock(src_ptl); > + if (is_pmd_migration_entry(src_pmdval)) { > + pmd_migration_entry_wait(src_mm, &src_pmdval); > + return -EAGAIN; > + } > + return -ENOENT; > + } > + > + src_page = pmd_page(src_pmdval); > + if (unlikely(!PageAnonExclusive(src_page))) { > + spin_unlock(src_ptl); > + return -EBUSY; > + } > + > + src_folio = page_folio(src_page); > + folio_get(src_folio); > + spin_unlock(src_ptl); > + > + /* preallocate dst_pgtable if needed */ > + if (dst_mm != src_mm) { > + dst_pgtable = pte_alloc_one(dst_mm); > + if (unlikely(!dst_pgtable)) { > + err = -ENOMEM; > + goto put_folio; > + } > + } else { > + dst_pgtable = NULL; > + } > + IIUC Lokesh's comment applies here, we probably need the flush_cache_range(), not for x86 but for the other ones.. cachetlb.rst: Next, we have the cache flushing interfaces. In general, when Linux is changing an existing virtual-->physical mapping to a new value, the sequence will be in one of the following forms:: 1) flush_cache_mm(mm); change_all_page_tables_of(mm); flush_tlb_mm(mm); 2) flush_cache_range(vma, start, end); change_range_of_page_tables(mm, start, end); flush_tlb_range(vma, start, end); 3) flush_cache_page(vma, addr, pfn); set_pte(pte_pointer, new_pte_val); flush_tlb_page(vma, addr); > + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr, > + src_addr + HPAGE_PMD_SIZE); > + mmu_notifier_invalidate_range_start(&range); > + > + folio_lock(src_folio); > + > + /* > + * split_huge_page walks the anon_vma chain without the page > + * lock. Serialize against it with the anon_vma lock, the page > + * lock is not enough. > + */ > + src_anon_vma = folio_get_anon_vma(src_folio); > + if (!src_anon_vma) { > + err = -EAGAIN; > + goto unlock_folio; > + } > + anon_vma_lock_write(src_anon_vma); > + > + dst_ptl = pmd_lockptr(dst_mm, dst_pmd); > + double_pt_lock(src_ptl, dst_ptl); > + if (unlikely(!pmd_same(*src_pmd, src_pmdval) || > + !pmd_same(*dst_pmd, dst_pmdval))) { > + double_pt_unlock(src_ptl, dst_ptl); > + err = -EAGAIN; > + goto put_anon_vma; > + } > + if (!PageAnonExclusive(&src_folio->page)) { > + double_pt_unlock(src_ptl, dst_ptl); > + err = -EBUSY; > + goto put_anon_vma; > + } > + > + BUG_ON(!folio_test_head(src_folio)); > + BUG_ON(!folio_test_anon(src_folio)); > + > + folio_move_anon_rmap(src_folio, dst_vma); > + WRITE_ONCE(src_folio->index, linear_page_index(dst_vma, dst_addr)); > + > + src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd); > + _dst_pmd = mk_huge_pmd(&src_folio->page, dst_vma->vm_page_prot); > + _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma); Last time the conclusion is we leverage can_change_pmd_writable(), no? > + set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd); > + > + src_pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd); > + if (dst_pgtable) { > + pgtable_trans_huge_deposit(dst_mm, dst_pmd, dst_pgtable); > + pte_free(src_mm, src_pgtable); > + dst_pgtable = NULL; > + > + mm_inc_nr_ptes(dst_mm); > + mm_dec_nr_ptes(src_mm); > + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); > + add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR); > + } else { > + pgtable_trans_huge_deposit(dst_mm, dst_pmd, src_pgtable); > + } > + double_pt_unlock(src_ptl, dst_ptl); > + > +put_anon_vma: > + anon_vma_unlock_write(src_anon_vma); > + put_anon_vma(src_anon_vma); > +unlock_folio: > + /* unblock rmap walks */ > + folio_unlock(src_folio); > + mmu_notifier_invalidate_range_end(&range); > + if (dst_pgtable) > + pte_free(dst_mm, dst_pgtable); > +put_folio: > + folio_put(src_folio); > + > + return err; > +} > +#endif /* CONFIG_USERFAULTFD */ > + > /* > * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise. > * > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > index 2b5c0321d96b..0c1ee7172852 100644 > --- a/mm/khugepaged.c > +++ b/mm/khugepaged.c > @@ -1136,6 +1136,9 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, > * Prevent all access to pagetables with the exception of > * gup_fast later handled by the ptep_clear_flush and the VM > * handled by the anon_vma lock + PG_lock. > + * > + * UFFDIO_MOVE is prevented to race as well thanks to the > + * mmap_lock. > */ > mmap_write_lock(mm); > result = hugepage_vma_revalidate(mm, address, true, &vma, cc); > diff --git a/mm/rmap.c b/mm/rmap.c > index f9ddc50269d2..a5919cac9a08 100644 > --- a/mm/rmap.c > +++ b/mm/rmap.c > @@ -490,6 +490,12 @@ void __init anon_vma_init(void) > * page_remove_rmap() that the anon_vma pointer from page->mapping is valid > * if there is a mapcount, we can dereference the anon_vma after observing > * those. > + * > + * NOTE: the caller should normally hold folio lock when calling this. If > + * not, the caller needs to double check the anon_vma didn't change after > + * taking the anon_vma lock for either read or write (UFFDIO_MOVE can modify it > + * concurrently without folio lock protection). See folio_lock_anon_vma_read() > + * which has already covered that, and comment above remap_pages(). > */ > struct anon_vma *folio_get_anon_vma(struct folio *folio) > { > diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c > index 96d9eae5c7cc..45ce1a8b8ab9 100644 > --- a/mm/userfaultfd.c > +++ b/mm/userfaultfd.c > @@ -842,3 +842,605 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, > mmap_read_unlock(dst_mm); > return err; > } > + > + > +void double_pt_lock(spinlock_t *ptl1, > + spinlock_t *ptl2) > + __acquires(ptl1) > + __acquires(ptl2) > +{ > + spinlock_t *ptl_tmp; > + > + if (ptl1 > ptl2) { > + /* exchange ptl1 and ptl2 */ > + ptl_tmp = ptl1; > + ptl1 = ptl2; > + ptl2 = ptl_tmp; > + } > + /* lock in virtual address order to avoid lock inversion */ > + spin_lock(ptl1); > + if (ptl1 != ptl2) > + spin_lock_nested(ptl2, SINGLE_DEPTH_NESTING); > + else > + __acquire(ptl2); > +} > + > +void double_pt_unlock(spinlock_t *ptl1, > + spinlock_t *ptl2) > + __releases(ptl1) > + __releases(ptl2) > +{ > + spin_unlock(ptl1); > + if (ptl1 != ptl2) > + spin_unlock(ptl2); > + else > + __release(ptl2); > +} > + > + > +static int remap_present_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, > + struct vm_area_struct *dst_vma, > + struct vm_area_struct *src_vma, > + unsigned long dst_addr, unsigned long src_addr, > + pte_t *dst_pte, pte_t *src_pte, > + pte_t orig_dst_pte, pte_t orig_src_pte, > + spinlock_t *dst_ptl, spinlock_t *src_ptl, > + struct folio *src_folio) > +{ > + double_pt_lock(dst_ptl, src_ptl); > + > + if (!pte_same(*src_pte, orig_src_pte) || > + !pte_same(*dst_pte, orig_dst_pte)) { > + double_pt_unlock(dst_ptl, src_ptl); > + return -EAGAIN; > + } > + if (folio_test_large(src_folio) || > + !PageAnonExclusive(&src_folio->page)) { > + double_pt_unlock(dst_ptl, src_ptl); > + return -EBUSY; > + } > + > + BUG_ON(!folio_test_anon(src_folio)); > + > + folio_move_anon_rmap(src_folio, dst_vma); > + WRITE_ONCE(src_folio->index, linear_page_index(dst_vma, dst_addr)); > + > + orig_src_pte = ptep_clear_flush(src_vma, src_addr, src_pte); > + orig_dst_pte = mk_pte(&src_folio->page, dst_vma->vm_page_prot); > + orig_dst_pte = maybe_mkwrite(pte_mkdirty(orig_dst_pte), dst_vma); can_change_pte_writable()? > + > + set_pte_at(dst_mm, dst_addr, dst_pte, orig_dst_pte); > + > + if (dst_mm != src_mm) { > + inc_mm_counter(dst_mm, MM_ANONPAGES); > + dec_mm_counter(src_mm, MM_ANONPAGES); > + } > + > + double_pt_unlock(dst_ptl, src_ptl); > + > + return 0; > +} > + > +static int remap_swap_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, > + unsigned long dst_addr, unsigned long src_addr, > + pte_t *dst_pte, pte_t *src_pte, > + pte_t orig_dst_pte, pte_t orig_src_pte, > + spinlock_t *dst_ptl, spinlock_t *src_ptl) > +{ > + if (!pte_swp_exclusive(orig_src_pte)) > + return -EBUSY; > + > + double_pt_lock(dst_ptl, src_ptl); > + > + if (!pte_same(*src_pte, orig_src_pte) || > + !pte_same(*dst_pte, orig_dst_pte)) { > + double_pt_unlock(dst_ptl, src_ptl); > + return -EAGAIN; > + } > + > + orig_src_pte = ptep_get_and_clear(src_mm, src_addr, src_pte); > + set_pte_at(dst_mm, dst_addr, dst_pte, orig_src_pte); > + > + if (dst_mm != src_mm) { > + inc_mm_counter(dst_mm, MM_SWAPENTS); > + dec_mm_counter(src_mm, MM_SWAPENTS); > + } > + > + double_pt_unlock(dst_ptl, src_ptl); > + > + return 0; > +} > + > +/* > + * The mmap_lock for reading is held by the caller. Just move the page > + * from src_pmd to dst_pmd if possible, and return true if succeeded > + * in moving the page. > + */ > +static int remap_pages_pte(struct mm_struct *dst_mm, > + struct mm_struct *src_mm, > + pmd_t *dst_pmd, > + pmd_t *src_pmd, > + struct vm_area_struct *dst_vma, > + struct vm_area_struct *src_vma, > + unsigned long dst_addr, > + unsigned long src_addr, > + __u64 mode) > +{ > + swp_entry_t entry; > + pte_t orig_src_pte, orig_dst_pte; > + pte_t src_folio_pte; > + spinlock_t *src_ptl, *dst_ptl; > + pte_t *src_pte = NULL; > + pte_t *dst_pte = NULL; > + > + struct folio *src_folio = NULL; > + struct anon_vma *src_anon_vma = NULL; > + struct mmu_notifier_range range; > + int err = 0; > + Same flush_cache_range(). > + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, > + src_addr, src_addr + PAGE_SIZE); > + mmu_notifier_invalidate_range_start(&range); > +retry: > + dst_pte = pte_offset_map_nolock(dst_mm, dst_pmd, dst_addr, &dst_ptl); > + > + /* Retry if a huge pmd materialized from under us */ > + if (unlikely(!dst_pte)) { > + err = -EAGAIN; > + goto out; > + } > + > + src_pte = pte_offset_map_nolock(src_mm, src_pmd, src_addr, &src_ptl); > + > + /* > + * We held the mmap_lock for reading so MADV_DONTNEED > + * can zap transparent huge pages under us, or the > + * transparent huge page fault can establish new > + * transparent huge pages under us. > + */ > + if (unlikely(!src_pte)) { > + err = -EAGAIN; > + goto out; > + } > + > + BUG_ON(pmd_none(*dst_pmd)); > + BUG_ON(pmd_none(*src_pmd)); > + BUG_ON(pmd_trans_huge(*dst_pmd)); > + BUG_ON(pmd_trans_huge(*src_pmd)); > + > + spin_lock(dst_ptl); > + orig_dst_pte = *dst_pte; > + spin_unlock(dst_ptl); > + if (!pte_none(orig_dst_pte)) { > + err = -EEXIST; > + goto out; > + } > + > + spin_lock(src_ptl); > + orig_src_pte = *src_pte; > + spin_unlock(src_ptl); > + if (pte_none(orig_src_pte)) { > + if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) > + err = -ENOENT; > + else /* nothing to do to remap a hole */ > + err = 0; > + goto out; > + } > + > + /* If PTE changed after we locked the folio them start over */ > + if (src_folio && unlikely(!pte_same(src_folio_pte, orig_src_pte))) { > + err = -EAGAIN; > + goto out; > + } > + > + if (pte_present(orig_src_pte)) { > + /* > + * Pin and lock both source folio and anon_vma. Since we are in > + * RCU read section, we can't block, so on contention have to > + * unmap the ptes, obtain the lock and retry. > + */ > + if (!src_folio) { > + struct folio *folio; > + > + /* > + * Pin the page while holding the lock to be sure the > + * page isn't freed under us > + */ > + spin_lock(src_ptl); > + if (!pte_same(orig_src_pte, *src_pte)) { > + spin_unlock(src_ptl); > + err = -EAGAIN; > + goto out; > + } > + > + folio = vm_normal_folio(src_vma, src_addr, orig_src_pte); > + if (!folio || folio_test_large(folio) || > + !PageAnonExclusive(&folio->page)) { > + spin_unlock(src_ptl); > + err = -EBUSY; > + goto out; > + } > + > + folio_get(folio); > + src_folio = folio; > + src_folio_pte = orig_src_pte; > + spin_unlock(src_ptl); > + > + if (!folio_trylock(src_folio)) { > + pte_unmap(&orig_src_pte); > + pte_unmap(&orig_dst_pte); > + src_pte = dst_pte = NULL; > + /* now we can block and wait */ > + folio_lock(src_folio); > + goto retry; > + } > + } > + > + if (!src_anon_vma) { > + /* > + * folio_referenced walks the anon_vma chain > + * without the folio lock. Serialize against it with > + * the anon_vma lock, the folio lock is not enough. > + */ > + src_anon_vma = folio_get_anon_vma(src_folio); > + if (!src_anon_vma) { > + /* page was unmapped from under us */ > + err = -EAGAIN; > + goto out; > + } > + if (!anon_vma_trylock_write(src_anon_vma)) { > + pte_unmap(&orig_src_pte); > + pte_unmap(&orig_dst_pte); > + src_pte = dst_pte = NULL; > + /* now we can block and wait */ > + anon_vma_lock_write(src_anon_vma); > + goto retry; > + } > + } > + > + err = remap_present_pte(dst_mm, src_mm, dst_vma, src_vma, > + dst_addr, src_addr, dst_pte, src_pte, > + orig_dst_pte, orig_src_pte, > + dst_ptl, src_ptl, src_folio); > + } else { > + entry = pte_to_swp_entry(orig_src_pte); > + if (non_swap_entry(entry)) { > + if (is_migration_entry(entry)) { > + pte_unmap(&orig_src_pte); > + pte_unmap(&orig_dst_pte); > + src_pte = dst_pte = NULL; > + migration_entry_wait(src_mm, src_pmd, > + src_addr); > + err = -EAGAIN; > + } else > + err = -EFAULT; > + goto out; > + } > + > + err = remap_swap_pte(dst_mm, src_mm, dst_addr, src_addr, > + dst_pte, src_pte, > + orig_dst_pte, orig_src_pte, > + dst_ptl, src_ptl); > + } > + > +out: > + if (src_anon_vma) { > + anon_vma_unlock_write(src_anon_vma); > + put_anon_vma(src_anon_vma); > + } > + if (src_folio) { > + folio_unlock(src_folio); > + folio_put(src_folio); > + } > + if (dst_pte) > + pte_unmap(dst_pte); > + if (src_pte) > + pte_unmap(src_pte); > + mmu_notifier_invalidate_range_end(&range); > + > + return err; > +} > + > +static int validate_remap_areas(struct vm_area_struct *src_vma, s/remap/move/ > + struct vm_area_struct *dst_vma) > +{ > + /* Only allow remapping if both have the same access and protection */ s/remap/move/ PS: I'll stop comment on renamings.. but please have a look over the whole patch.. > + if ((src_vma->vm_flags & VM_ACCESS_FLAGS) != (dst_vma->vm_flags & VM_ACCESS_FLAGS) || > + pgprot_val(src_vma->vm_page_prot) != pgprot_val(dst_vma->vm_page_prot)) > + return -EINVAL; > + > + /* Only allow remapping if both are mlocked or both aren't */ > + if ((src_vma->vm_flags & VM_LOCKED) != (dst_vma->vm_flags & VM_LOCKED)) > + return -EINVAL; > + > + if (!(src_vma->vm_flags & VM_WRITE) || !(dst_vma->vm_flags & VM_WRITE)) > + return -EINVAL; > + > + /* > + * Be strict and only allow remap_pages if either the src or > + * dst range is registered in the userfaultfd to prevent > + * userland errors going unnoticed. As far as the VM > + * consistency is concerned, it would be perfectly safe to > + * remove this check, but there's no useful usage for > + * remap_pages ouside of userfaultfd registered ranges. This > + * is after all why it is an ioctl belonging to the > + * userfaultfd and not a syscall. > + * > + * Allow both vmas to be registered in the userfaultfd, just > + * in case somebody finds a way to make such a case useful. > + * Normally only one of the two vmas would be registered in > + * the userfaultfd. > + */ > + if (!dst_vma->vm_userfaultfd_ctx.ctx && > + !src_vma->vm_userfaultfd_ctx.ctx) > + return -EINVAL; When rethinking this, I'm wondering whether we should make it strict. The problem is, src/dst is not equal in this case. We got dst_mm from the uffd context passed in, so we know at least dst_mm has something registered onto this uffd we're working on, no matter it's the dst_vma or not. It'll be weird if dst_vma is not registered to the uffd context we're using right now.. Then, can we have a case where src vma is registered with uffd, then dest vma is not? It sounds really weird, since dest mm needs to be registered to uffd anyway. It sounds to me the interface is a bit loose and unclear. I'm thinking a more reasonable way to safe check this: we model UFFDIO_MOVE the same way as UFFDIO_COPY/CONTINUE/... where we always assumed the uffd context is the target (uffd registration required), while current mm is always the source (no uffd registration required). It also means "dropping current page" does not require privilege (here UFFDIO_MOVE is the same as DONTNEED, per discussed), while "installing a page into the pgtable" does require some privilege (where privilege granted by the uffd itself). Then it means something like: if (!dst_vma->vm_userfaultfd_ctx.ctx || dst_vma->vm_userfaultfd_ctx.ctx != ctx) return -EINVAL; Here ctx should be passed over from upper. What do you think? PS: it seems current uffd ioctls are not double checking the vma's ctx is the same as the ctx that the ioctl is operating on.. but it seems we should do that for all. I'll think more about it.. > + > + /* > + * FIXME: only allow remapping across anonymous vmas, > + * tmpfs should be added. > + */ > + if (!vma_is_anonymous(src_vma) || !vma_is_anonymous(dst_vma)) > + return -EINVAL; > + > + /* > + * Ensure the dst_vma has a anon_vma or this page > + * would get a NULL anon_vma when moved in the > + * dst_vma. > + */ > + if (unlikely(anon_vma_prepare(dst_vma))) > + return -ENOMEM; > + > + return 0; > +} > + > +/** > + * remap_pages - remap arbitrary anonymous pages of an existing vma > + * @dst_start: start of the destination virtual memory range > + * @src_start: start of the source virtual memory range > + * @len: length of the virtual memory range > + * > + * remap_pages() remaps arbitrary anonymous pages atomically in zero > + * copy. It only works on non shared anonymous pages because those can > + * be relocated without generating non linear anon_vmas in the rmap > + * code. > + * > + * It provides a zero copy mechanism to handle userspace page faults. > + * The source vma pages should have mapcount == 1, which can be > + * enforced by using madvise(MADV_DONTFORK) on src vma. > + * > + * The thread receiving the page during the userland page fault > + * will receive the faulting page in the source vma through the network, > + * storage or any other I/O device (MADV_DONTFORK in the source vma > + * avoids remap_pages() to fail with -EBUSY if the process forks before > + * remap_pages() is called), then it will call remap_pages() to map the > + * page in the faulting address in the destination vma. > + * > + * This userfaultfd command works purely via pagetables, so it's the > + * most efficient way to move physical non shared anonymous pages > + * across different virtual addresses. Unlike mremap()/mmap()/munmap() > + * it does not create any new vmas. The mapping in the destination > + * address is atomic. > + * > + * It only works if the vma protection bits are identical from the > + * source and destination vma. > + * > + * It can remap non shared anonymous pages within the same vma too. > + * > + * If the source virtual memory range has any unmapped holes, or if > + * the destination virtual memory range is not a whole unmapped hole, > + * remap_pages() will fail respectively with -ENOENT or -EEXIST. This > + * provides a very strict behavior to avoid any chance of memory > + * corruption going unnoticed if there are userland race conditions. > + * Only one thread should resolve the userland page fault at any given > + * time for any given faulting address. This means that if two threads > + * try to both call remap_pages() on the same destination address at the > + * same time, the second thread will get an explicit error from this > + * command. > + * > + * The command retval will return "len" is successful. The command > + * however can be interrupted by fatal signals or errors. If > + * interrupted it will return the number of bytes successfully > + * remapped before the interruption if any, or the negative error if > + * none. It will never return zero. Either it will return an error or > + * an amount of bytes successfully moved. If the retval reports a > + * "short" remap, the remap_pages() command should be repeated by > + * userland with src+retval, dst+reval, len-retval if it wants to know > + * about the error that interrupted it. > + * > + * The UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES flag can be specified to > + * prevent -ENOENT errors to materialize if there are holes in the > + * source virtual range that is being remapped. The holes will be > + * accounted as successfully remapped in the retval of the > + * command. This is mostly useful to remap hugepage naturally aligned > + * virtual regions without knowing if there are transparent hugepage > + * in the regions or not, but preventing the risk of having to split > + * the hugepmd during the remap. > + * > + * If there's any rmap walk that is taking the anon_vma locks without > + * first obtaining the folio lock (the only current instance is > + * folio_referenced), they will have to verify if the folio->mapping > + * has changed after taking the anon_vma lock. If it changed they > + * should release the lock and retry obtaining a new anon_vma, because > + * it means the anon_vma was changed by remap_pages() before the lock > + * could be obtained. This is the only additional complexity added to > + * the rmap code to provide this anonymous page remapping functionality. > + */ > +ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm, > + unsigned long dst_start, unsigned long src_start, > + unsigned long len, __u64 mode) > +{ > + struct vm_area_struct *src_vma, *dst_vma; > + unsigned long src_addr, dst_addr; > + pmd_t *src_pmd, *dst_pmd; > + long err = -EINVAL; > + ssize_t moved = 0; > + > + /* > + * Sanitize the command parameters: > + */ > + BUG_ON(src_start & ~PAGE_MASK); > + BUG_ON(dst_start & ~PAGE_MASK); > + BUG_ON(len & ~PAGE_MASK); > + > + /* Does the address range wrap, or is the span zero-sized? */ > + BUG_ON(src_start + len <= src_start); > + BUG_ON(dst_start + len <= dst_start); > + > + /* > + * Because these are read sempahores there's no risk of lock > + * inversion. > + */ > + mmap_read_lock(dst_mm); > + if (dst_mm != src_mm) > + mmap_read_lock(src_mm); > + > + /* > + * Make sure the vma is not shared, that the src and dst remap > + * ranges are both valid and fully within a single existing > + * vma. > + */ > + src_vma = find_vma(src_mm, src_start); > + if (!src_vma || (src_vma->vm_flags & VM_SHARED)) > + goto out; > + if (src_start < src_vma->vm_start || > + src_start + len > src_vma->vm_end) > + goto out; > + > + dst_vma = find_vma(dst_mm, dst_start); > + if (!dst_vma || (dst_vma->vm_flags & VM_SHARED)) > + goto out; > + if (dst_start < dst_vma->vm_start || > + dst_start + len > dst_vma->vm_end) > + goto out; > + > + err = validate_remap_areas(src_vma, dst_vma); > + if (err) > + goto out; > + > + for (src_addr = src_start, dst_addr = dst_start; > + src_addr < src_start + len;) { > + spinlock_t *ptl; > + pmd_t dst_pmdval; > + unsigned long step_size; > + > + BUG_ON(dst_addr >= dst_start + len); > + /* > + * Below works because anonymous area would not have a > + * transparent huge PUD. If file-backed support is added, > + * that case would need to be handled here. > + */ > + src_pmd = mm_find_pmd(src_mm, src_addr); > + if (unlikely(!src_pmd)) { > + if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) { > + err = -ENOENT; > + break; > + } > + src_pmd = mm_alloc_pmd(src_mm, src_addr); > + if (unlikely(!src_pmd)) { > + err = -ENOMEM; > + break; > + } > + } > + dst_pmd = mm_alloc_pmd(dst_mm, dst_addr); > + if (unlikely(!dst_pmd)) { > + err = -ENOMEM; > + break; > + } > + > + dst_pmdval = pmdp_get_lockless(dst_pmd); > + /* > + * If the dst_pmd is mapped as THP don't override it and just > + * be strict. If dst_pmd changes into TPH after this check, the > + * remap_pages_huge_pmd() will detect the change and retry > + * while remap_pages_pte() will detect the change and fail. > + */ > + if (unlikely(pmd_trans_huge(dst_pmdval))) { > + err = -EEXIST; > + break; > + } > + > + ptl = pmd_trans_huge_lock(src_pmd, src_vma); > + if (ptl) { > + if (pmd_devmap(*src_pmd)) { > + spin_unlock(ptl); > + err = -ENOENT; > + break; > + } > + > + /* > + * Check if we can move the pmd without > + * splitting it. First check the address > + * alignment to be the same in src/dst. These > + * checks don't actually need the PT lock but > + * it's good to do it here to optimize this > + * block away at build time if > + * CONFIG_TRANSPARENT_HUGEPAGE is not set. > + */ > + if ((src_addr & ~HPAGE_PMD_MASK) || (dst_addr & ~HPAGE_PMD_MASK) || > + src_start + len - src_addr < HPAGE_PMD_SIZE || !pmd_none(dst_pmdval)) { > + spin_unlock(ptl); > + split_huge_pmd(src_vma, src_pmd, src_addr); > + continue; > + } > + > + err = remap_pages_huge_pmd(dst_mm, src_mm, > + dst_pmd, src_pmd, > + dst_pmdval, > + dst_vma, src_vma, > + dst_addr, src_addr); > + step_size = HPAGE_PMD_SIZE; > + } else { > + if (pmd_none(*src_pmd)) { > + if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) { > + err = -ENOENT; > + break; > + } > + if (unlikely(__pte_alloc(src_mm, src_pmd))) { > + err = -ENOMEM; > + break; > + } > + } > + > + if (unlikely(pte_alloc(dst_mm, dst_pmd))) { > + err = -ENOMEM; > + break; > + } > + > + err = remap_pages_pte(dst_mm, src_mm, > + dst_pmd, src_pmd, > + dst_vma, src_vma, > + dst_addr, src_addr, > + mode); > + step_size = PAGE_SIZE; > + } > + > + cond_resched(); > + > + if (fatal_signal_pending(current)) { > + /* Do not override an error */ > + if (!err || err == -EAGAIN) > + err = -EINTR; > + break; > + } > + > + if (err) { > + if (err == -EAGAIN) > + continue; > + break; > + } > + > + /* Proceed to the next page */ > + dst_addr += step_size; > + src_addr += step_size; > + moved += step_size; > + } > + > +out: > + mmap_read_unlock(dst_mm); > + if (dst_mm != src_mm) > + mmap_read_unlock(src_mm); > + BUG_ON(moved < 0); > + BUG_ON(err > 0); > + BUG_ON(!moved && !err); > + return moved ? moved : err; > +} > -- > 2.42.0.609.gbb76f46606-goog >
On 12.10.23 22:11, Peter Xu wrote: > On Mon, Oct 09, 2023 at 05:29:08PM +0100, Lokesh Gidra wrote: >> On Mon, Oct 9, 2023 at 5:24 PM David Hildenbrand <david@redhat.com> wrote: >>> >>> On 09.10.23 18:21, Suren Baghdasaryan wrote: >>>> On Mon, Oct 9, 2023 at 7:38 AM David Hildenbrand <david@redhat.com> wrote: >>>>> >>>>> On 09.10.23 08:42, Suren Baghdasaryan wrote: >>>>>> From: Andrea Arcangeli <aarcange@redhat.com> >>>>>> >>>>>> Implement the uABI of UFFDIO_MOVE ioctl. >>>>>> UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application >>>>>> needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are >>>>>> available (in userspace) for recycling, as is usually the case in heap >>>>>> compaction algorithms, then we can avoid the page allocation and memcpy >>>>>> (done by UFFDIO_COPY). Also, since the pages are recycled in the >>>>>> userspace, we avoid the need to release (via madvise) the pages back to >>>>>> the kernel [2]. >>>>>> We see over 40% reduction (on a Google pixel 6 device) in the compacting >>>>>> thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was >>>>>> measured using a benchmark that emulates a heap compaction implementation >>>>>> using userfaultfd (to allow concurrent accesses by application threads). >>>>>> More details of the usecase are explained in [2]. >>>>>> Furthermore, UFFDIO_MOVE enables moving swapped-out pages without >>>>>> touching them within the same vma. Today, it can only be done by mremap, >>>>>> however it forces splitting the vma. >>>>>> >>>>>> [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ >>>>>> [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ >>>>>> >>>>>> Update for the ioctl_userfaultfd(2) manpage: >>>>>> >>>>>> UFFDIO_MOVE >>>>>> (Since Linux xxx) Move a continuous memory chunk into the >>>>>> userfault registered range and optionally wake up the blocked >>>>>> thread. The source and destination addresses and the number of >>>>>> bytes to move are specified by the src, dst, and len fields of >>>>>> the uffdio_move structure pointed to by argp: >>>>>> >>>>>> struct uffdio_move { >>>>>> __u64 dst; /* Destination of move */ >>>>>> __u64 src; /* Source of move */ >>>>>> __u64 len; /* Number of bytes to move */ >>>>>> __u64 mode; /* Flags controlling behavior of move */ >>>>>> __s64 move; /* Number of bytes moved, or negated error */ >>>>>> }; >>>>>> >>>>>> The following value may be bitwise ORed in mode to change the >>>>>> behavior of the UFFDIO_MOVE operation: >>>>>> >>>>>> UFFDIO_MOVE_MODE_DONTWAKE >>>>>> Do not wake up the thread that waits for page-fault >>>>>> resolution >>>>>> >>>>>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES >>>>>> Allow holes in the source virtual range that is being moved. >>>>>> When not specified, the holes will result in ENOENT error. >>>>>> When specified, the holes will be accounted as successfully >>>>>> moved memory. This is mostly useful to move hugepage aligned >>>>>> virtual regions without knowing if there are transparent >>>>>> hugepages in the regions or not, but preventing the risk of >>>>>> having to split the hugepage during the operation. >>>>>> >>>>>> The move field is used by the kernel to return the number of >>>>>> bytes that was actually moved, or an error (a negated errno- >>>>>> style value). If the value returned in move doesn't match the >>>>>> value that was specified in len, the operation fails with the >>>>>> error EAGAIN. The move field is output-only; it is not read by >>>>>> the UFFDIO_MOVE operation. >>>>>> >>>>>> The operation may fail for various reasons. Usually, remapping of >>>>>> pages that are not exclusive to the given process fail; once KSM >>>>>> might deduplicate pages or fork() COW-shares pages during fork() >>>>>> with child processes, they are no longer exclusive. Further, the >>>>>> kernel might only perform lightweight checks for detecting whether >>>>>> the pages are exclusive, and return -EBUSY in case that check fails. >>>>>> To make the operation more likely to succeed, KSM should be >>>>>> disabled, fork() should be avoided or MADV_DONTFORK should be >>>>>> configured for the source VMA before fork(). >>>>>> >>>>>> This ioctl(2) operation returns 0 on success. In this case, the >>>>>> entire area was moved. On error, -1 is returned and errno is >>>>>> set to indicate the error. Possible errors include: >>>>>> >>>>>> EAGAIN The number of bytes moved (i.e., the value returned in >>>>>> the move field) does not equal the value that was >>>>>> specified in the len field. >>>>>> >>>>>> EINVAL Either dst or len was not a multiple of the system page >>>>>> size, or the range specified by src and len or dst and len >>>>>> was invalid. >>>>>> >>>>>> EINVAL An invalid bit was specified in the mode field. >>>>>> >>>>>> ENOENT >>>>>> The source virtual memory range has unmapped holes and >>>>>> UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. >>>>>> >>>>>> EEXIST >>>>>> The destination virtual memory range is fully or partially >>>>>> mapped. >>>>>> >>>>>> EBUSY >>>>>> The pages in the source virtual memory range are not >>>>>> exclusive to the process. The kernel might only perform >>>>>> lightweight checks for detecting whether the pages are >>>>>> exclusive. To make the operation more likely to succeed, >>>>>> KSM should be disabled, fork() should be avoided or >>>>>> MADV_DONTFORK should be configured for the source virtual >>>>>> memory area before fork(). >>>>>> >>>>>> ENOMEM Allocating memory needed for the operation failed. >>>>>> >>>>>> ESRCH >>>>>> The faulting process has exited at the time of a >>>>>> UFFDIO_MOVE operation. >>>>>> >>>>> >>>>> A general comment simply because I realized that just now: does anything >>>>> speak against limiting the operations now to a single MM? >>>>> >>>>> The use cases I heard so far don't need it. If ever required, we could >>>>> consider extending it. >>>>> >>>>> Let's reduce complexity and KIS unless really required. >>>> >>>> Let me check if there are use cases that require moves between MMs. >>>> Andrea seems to have put considerable effort to make it work between >>>> MMs and it would be a pity to lose that. I can send a follow-up patch >>>> to recover that functionality and even if it does not get merged, it >>>> can be used in the future as a reference. But first let me check if we >>>> can drop it. >> >> For the compaction use case that we have it's fine to limit it to >> single MM. However, for general use I think Peter will have a better >> idea. > Hi Peter, > I used to have the same thought with David on whether we can simplify the > design to e.g. limit it to single mm. Then I found that the trickiest is > actually patch 1 together with the anon_vma manipulations, and the problem > is that's not avoidable even if we restrict the api to apply on single mm. > > What else we can benefit from single mm? One less mmap read lock, but > probably that's all we can get; IIUC we need to keep most of the rest of > the code, e.g. pgtable walks, double pgtable lockings, etc. No existing mechanisms move anon pages between unrelated processes, that naturally makes me nervous if we're doing it "just because we can". > > Actually, even though I have no solid clue, but I had a feeling that there > can be some interesting way to leverage this across-mm movement, while > keeping things all safe (by e.g. elaborately requiring other proc to create > uffd and deliver to this proc). Okay, but no real use cases yet. > > Considering Andrea's original version already contains those bits and all > above, I'd vote that we go ahead with supporting two MMs. You can do nasty things with that, as it stands, on the upstream codebase. If you pin the page in src_mm and move it to dst_mm, you successfully broke an invariant that "exclusive" means "no other references from other processes". That page is marked exclusive but it is, in fact, not exclusive. Once you achieved that, you can easily have src_mm not have MMF_HAS_PINNED, so you can just COW-share that page. Now you successfully broke the invariant that COW-shared pages must not be pinned. And you can even trigger VM_BUG_ONs, like in sanity_check_pinned_pages(). Can it all be fixed? Sure, with more complexity. For something without clear motivation, I'll have to pass. Once there is real demand, we can revisit it and explore what else we would have to take care of (I don't know how memcg behaves when moving between completely unrelated processes, maybe that works as expected, I don't know and I have no time to spare on reviewing features with no real use cases) and announce it as a new feature. Note: that (with only reading the documentation) it also kept me wondering how the MMs are even implied from struct uffdio_move { __u64 dst; /* Destination of move */ __u64 src; /* Source of move */ __u64 len; /* Number of bytes to move */ __u64 mode; /* Flags controlling behavior of move */ __s64 move; /* Number of bytes moved, or negated error */ }; That probably has to be documented as well, in which address space dst and src reside.
On Fri, Oct 13, 2023 at 11:56:31AM +0200, David Hildenbrand wrote: > Hi Peter, Hi, David, > > > I used to have the same thought with David on whether we can simplify the > > design to e.g. limit it to single mm. Then I found that the trickiest is > > actually patch 1 together with the anon_vma manipulations, and the problem > > is that's not avoidable even if we restrict the api to apply on single mm. > > > > What else we can benefit from single mm? One less mmap read lock, but > > probably that's all we can get; IIUC we need to keep most of the rest of > > the code, e.g. pgtable walks, double pgtable lockings, etc. > > No existing mechanisms move anon pages between unrelated processes, that > naturally makes me nervous if we're doing it "just because we can". IMHO that's also the potential, when guarded with userfaultfd descriptor being shared between two processes. See below with more comment on the raised concerns. > > > > > Actually, even though I have no solid clue, but I had a feeling that there > > can be some interesting way to leverage this across-mm movement, while > > keeping things all safe (by e.g. elaborately requiring other proc to create > > uffd and deliver to this proc). > > Okay, but no real use cases yet. I can provide a "not solid" example. I didn't mention it because it's really something that just popped into my mind when thinking cross-mm, so I never discussed with anyone yet nor shared it anywhere. Consider VM live upgrade in a generic form (e.g., no VFIO), we can do that very efficiently with shmem or hugetlbfs, but not yet anonymous. We can do extremely efficient postcopy live upgrade now with anonymous if with REMAP. Basically I see it a potential way of moving memory efficiently especially with thp. > > > > > Considering Andrea's original version already contains those bits and all > > above, I'd vote that we go ahead with supporting two MMs. > > You can do nasty things with that, as it stands, on the upstream codebase. > > If you pin the page in src_mm and move it to dst_mm, you successfully broke > an invariant that "exclusive" means "no other references from other > processes". That page is marked exclusive but it is, in fact, not exclusive. It is still exclusive to the dst mm? I see your point, but I think you're taking exclusiveness altogether with pinning, and IMHO that may not be always necessary? > > Once you achieved that, you can easily have src_mm not have MMF_HAS_PINNED, (I suppose you meant dst_mm here) > so you can just COW-share that page. Now you successfully broke the > invariant that COW-shared pages must not be pinned. And you can even trigger > VM_BUG_ONs, like in sanity_check_pinned_pages(). Yeah, that's really unfortunate. But frankly, I don't think it's the fault of this new feature, but the rest. Let's imagine if the MMF_HAS_PINNED wasn't proposed as a per-mm flag, but per-vma, which I don't see why we can't because it's simply a hint so far. Then if we apply the same rule here, UFFDIO_REMAP won't even work for single-mm as long as cross-vma. Then UFFDIO_REMAP as a whole feature will be NACKed simply because of this.. And I don't think anyone can guarantee a per-vma MMF_HAS_PINNED can never happen, or any further change to pinning solution that may affect this. So far it just looks unsafe to remap a pin page to me. I don't have a good suggestion here if this is a risk.. I'd think it risky then to do REMAP over pinned pages no matter cross-mm or single-mm. It means probably we just rule them out: folio_maybe_dma_pinned() may not even be enough to be safe with fast-gup. We may need page_needs_cow_for_dma() with proper write_protect_seq no matter cross-mm or single-mm? > > Can it all be fixed? Sure, with more complexity. For something without clear > motivation, I'll have to pass. I think what you raised is a valid concern, but IMHO it's better fixed no matter cross-mm or single-mm. What do you think? In general, pinning lose its whole point here to me for an userspace either if it DONTNEEDs it or REMAP it. What would be great to do here is we unpin it upon DONTNEED/REMAP/whatever drops the page, because it loses its coherency anyway, IMHO. > > Once there is real demand, we can revisit it and explore what else we would > have to take care of (I don't know how memcg behaves when moving between > completely unrelated processes, maybe that works as expected, I don't know > and I have no time to spare on reviewing features with no real use cases) > and announce it as a new feature. Good point. memcg is probably needed.. So you reminded me to do a more thorough review against zap/fault paths, I think what's missing are (besides page pinning): - mem_cgroup_charge()/mem_cgroup_uncharge(): (side note: I think folio_throttle_swaprate() is only for when allocating new pages, so not needed here) - check_stable_address_space() (under pgtable lock) - tlb flush Hmm???????????????? I can't see anywhere we did tlb flush, batched or not, either single-mm or cross-mm should need it. Is this missing? > > > Note: that (with only reading the documentation) it also kept me wondering > how the MMs are even implied from > > struct uffdio_move { > __u64 dst; /* Destination of move */ > __u64 src; /* Source of move */ > __u64 len; /* Number of bytes to move */ > __u64 mode; /* Flags controlling behavior of move */ > __s64 move; /* Number of bytes moved, or negated error */ > }; > > That probably has to be documented as well, in which address space dst and > src reside. Agreed, some better documentation will never hurt. Dst should be in the mm address space that was bound to the userfault descriptor. Src should be in the current mm address space. Thanks,
On Fri, Oct 13, 2023 at 9:08 AM Peter Xu <peterx@redhat.com> wrote: > > On Fri, Oct 13, 2023 at 11:56:31AM +0200, David Hildenbrand wrote: > > Hi Peter, > > Hi, David, > > > > > > I used to have the same thought with David on whether we can simplify the > > > design to e.g. limit it to single mm. Then I found that the trickiest is > > > actually patch 1 together with the anon_vma manipulations, and the problem > > > is that's not avoidable even if we restrict the api to apply on single mm. > > > > > > What else we can benefit from single mm? One less mmap read lock, but > > > probably that's all we can get; IIUC we need to keep most of the rest of > > > the code, e.g. pgtable walks, double pgtable lockings, etc. > > > > No existing mechanisms move anon pages between unrelated processes, that > > naturally makes me nervous if we're doing it "just because we can". > > IMHO that's also the potential, when guarded with userfaultfd descriptor > being shared between two processes. > > See below with more comment on the raised concerns. > > > > > > > > > Actually, even though I have no solid clue, but I had a feeling that there > > > can be some interesting way to leverage this across-mm movement, while > > > keeping things all safe (by e.g. elaborately requiring other proc to create > > > uffd and deliver to this proc). > > > > Okay, but no real use cases yet. > > I can provide a "not solid" example. I didn't mention it because it's > really something that just popped into my mind when thinking cross-mm, so I > never discussed with anyone yet nor shared it anywhere. > > Consider VM live upgrade in a generic form (e.g., no VFIO), we can do that > very efficiently with shmem or hugetlbfs, but not yet anonymous. We can do > extremely efficient postcopy live upgrade now with anonymous if with REMAP. > > Basically I see it a potential way of moving memory efficiently especially > with thp. > > > > > > > > > Considering Andrea's original version already contains those bits and all > > > above, I'd vote that we go ahead with supporting two MMs. > > > > You can do nasty things with that, as it stands, on the upstream codebase. > > > > If you pin the page in src_mm and move it to dst_mm, you successfully broke > > an invariant that "exclusive" means "no other references from other > > processes". That page is marked exclusive but it is, in fact, not exclusive. > > It is still exclusive to the dst mm? I see your point, but I think you're > taking exclusiveness altogether with pinning, and IMHO that may not be > always necessary? > > > > > Once you achieved that, you can easily have src_mm not have MMF_HAS_PINNED, > > (I suppose you meant dst_mm here) > > > so you can just COW-share that page. Now you successfully broke the > > invariant that COW-shared pages must not be pinned. And you can even trigger > > VM_BUG_ONs, like in sanity_check_pinned_pages(). > > Yeah, that's really unfortunate. But frankly, I don't think it's the fault > of this new feature, but the rest. > > Let's imagine if the MMF_HAS_PINNED wasn't proposed as a per-mm flag, but > per-vma, which I don't see why we can't because it's simply a hint so far. > Then if we apply the same rule here, UFFDIO_REMAP won't even work for > single-mm as long as cross-vma. Then UFFDIO_REMAP as a whole feature will > be NACKed simply because of this.. > > And I don't think anyone can guarantee a per-vma MMF_HAS_PINNED can never > happen, or any further change to pinning solution that may affect this. So > far it just looks unsafe to remap a pin page to me. > > I don't have a good suggestion here if this is a risk.. I'd think it risky > then to do REMAP over pinned pages no matter cross-mm or single-mm. It > means probably we just rule them out: folio_maybe_dma_pinned() may not even > be enough to be safe with fast-gup. We may need page_needs_cow_for_dma() > with proper write_protect_seq no matter cross-mm or single-mm? > > > > > Can it all be fixed? Sure, with more complexity. For something without clear > > motivation, I'll have to pass. > > I think what you raised is a valid concern, but IMHO it's better fixed no > matter cross-mm or single-mm. What do you think? > > In general, pinning lose its whole point here to me for an userspace either > if it DONTNEEDs it or REMAP it. What would be great to do here is we unpin > it upon DONTNEED/REMAP/whatever drops the page, because it loses its > coherency anyway, IMHO. > > > > > Once there is real demand, we can revisit it and explore what else we would > > have to take care of (I don't know how memcg behaves when moving between > > completely unrelated processes, maybe that works as expected, I don't know > > and I have no time to spare on reviewing features with no real use cases) > > and announce it as a new feature. > > Good point. memcg is probably needed.. > > So you reminded me to do a more thorough review against zap/fault paths, I > think what's missing are (besides page pinning): > > - mem_cgroup_charge()/mem_cgroup_uncharge(): > > (side note: I think folio_throttle_swaprate() is only for when > allocating new pages, so not needed here) > > - check_stable_address_space() (under pgtable lock) > > - tlb flush > > Hmm???????????????? I can't see anywhere we did tlb flush, batched or > not, either single-mm or cross-mm should need it. Is this missing? > IIUC, ptep_clear_flush() flushes tlb entry. So I think we are doing unbatched flushing. Possibly a nice performance improvement later on would be to try doing it batched. Suren can throw more light on it. One thing I was wondering is don't we need cache flush for the src pages? mremap's move_page_tables() does it. IMHO, it's required here as well. > > > > > > Note: that (with only reading the documentation) it also kept me wondering > > how the MMs are even implied from > > > > struct uffdio_move { > > __u64 dst; /* Destination of move */ > > __u64 src; /* Source of move */ > > __u64 len; /* Number of bytes to move */ > > __u64 mode; /* Flags controlling behavior of move */ > > __s64 move; /* Number of bytes moved, or negated error */ > > }; > > > > That probably has to be documented as well, in which address space dst and > > src reside. > > Agreed, some better documentation will never hurt. Dst should be in the mm > address space that was bound to the userfault descriptor. Src should be in > the current mm address space. > > Thanks, > > -- > Peter Xu >
On Fri, Oct 13, 2023 at 09:49:10AM -0700, Lokesh Gidra wrote: > On Fri, Oct 13, 2023 at 9:08 AM Peter Xu <peterx@redhat.com> wrote: > > > > On Fri, Oct 13, 2023 at 11:56:31AM +0200, David Hildenbrand wrote: > > > Hi Peter, > > > > Hi, David, > > > > > > > > > I used to have the same thought with David on whether we can simplify the > > > > design to e.g. limit it to single mm. Then I found that the trickiest is > > > > actually patch 1 together with the anon_vma manipulations, and the problem > > > > is that's not avoidable even if we restrict the api to apply on single mm. > > > > > > > > What else we can benefit from single mm? One less mmap read lock, but > > > > probably that's all we can get; IIUC we need to keep most of the rest of > > > > the code, e.g. pgtable walks, double pgtable lockings, etc. > > > > > > No existing mechanisms move anon pages between unrelated processes, that > > > naturally makes me nervous if we're doing it "just because we can". > > > > IMHO that's also the potential, when guarded with userfaultfd descriptor > > being shared between two processes. > > > > See below with more comment on the raised concerns. > > > > > > > > > > > > > Actually, even though I have no solid clue, but I had a feeling that there > > > > can be some interesting way to leverage this across-mm movement, while > > > > keeping things all safe (by e.g. elaborately requiring other proc to create > > > > uffd and deliver to this proc). > > > > > > Okay, but no real use cases yet. > > > > I can provide a "not solid" example. I didn't mention it because it's > > really something that just popped into my mind when thinking cross-mm, so I > > never discussed with anyone yet nor shared it anywhere. > > > > Consider VM live upgrade in a generic form (e.g., no VFIO), we can do that > > very efficiently with shmem or hugetlbfs, but not yet anonymous. We can do > > extremely efficient postcopy live upgrade now with anonymous if with REMAP. > > > > Basically I see it a potential way of moving memory efficiently especially > > with thp. > > > > > > > > > > > > > Considering Andrea's original version already contains those bits and all > > > > above, I'd vote that we go ahead with supporting two MMs. > > > > > > You can do nasty things with that, as it stands, on the upstream codebase. > > > > > > If you pin the page in src_mm and move it to dst_mm, you successfully broke > > > an invariant that "exclusive" means "no other references from other > > > processes". That page is marked exclusive but it is, in fact, not exclusive. > > > > It is still exclusive to the dst mm? I see your point, but I think you're > > taking exclusiveness altogether with pinning, and IMHO that may not be > > always necessary? > > > > > > > > Once you achieved that, you can easily have src_mm not have MMF_HAS_PINNED, > > > > (I suppose you meant dst_mm here) > > > > > so you can just COW-share that page. Now you successfully broke the > > > invariant that COW-shared pages must not be pinned. And you can even trigger > > > VM_BUG_ONs, like in sanity_check_pinned_pages(). > > > > Yeah, that's really unfortunate. But frankly, I don't think it's the fault > > of this new feature, but the rest. > > > > Let's imagine if the MMF_HAS_PINNED wasn't proposed as a per-mm flag, but > > per-vma, which I don't see why we can't because it's simply a hint so far. > > Then if we apply the same rule here, UFFDIO_REMAP won't even work for > > single-mm as long as cross-vma. Then UFFDIO_REMAP as a whole feature will > > be NACKed simply because of this.. > > > > And I don't think anyone can guarantee a per-vma MMF_HAS_PINNED can never > > happen, or any further change to pinning solution that may affect this. So > > far it just looks unsafe to remap a pin page to me. > > > > I don't have a good suggestion here if this is a risk.. I'd think it risky > > then to do REMAP over pinned pages no matter cross-mm or single-mm. It > > means probably we just rule them out: folio_maybe_dma_pinned() may not even > > be enough to be safe with fast-gup. We may need page_needs_cow_for_dma() > > with proper write_protect_seq no matter cross-mm or single-mm? > > > > > > > > Can it all be fixed? Sure, with more complexity. For something without clear > > > motivation, I'll have to pass. > > > > I think what you raised is a valid concern, but IMHO it's better fixed no > > matter cross-mm or single-mm. What do you think? > > > > In general, pinning lose its whole point here to me for an userspace either > > if it DONTNEEDs it or REMAP it. What would be great to do here is we unpin > > it upon DONTNEED/REMAP/whatever drops the page, because it loses its > > coherency anyway, IMHO. > > > > > > > > Once there is real demand, we can revisit it and explore what else we would > > > have to take care of (I don't know how memcg behaves when moving between > > > completely unrelated processes, maybe that works as expected, I don't know > > > and I have no time to spare on reviewing features with no real use cases) > > > and announce it as a new feature. > > > > Good point. memcg is probably needed.. > > > > So you reminded me to do a more thorough review against zap/fault paths, I > > think what's missing are (besides page pinning): > > > > - mem_cgroup_charge()/mem_cgroup_uncharge(): > > > > (side note: I think folio_throttle_swaprate() is only for when > > allocating new pages, so not needed here) > > > > - check_stable_address_space() (under pgtable lock) > > > > - tlb flush > > > > Hmm???????????????? I can't see anywhere we did tlb flush, batched or > > not, either single-mm or cross-mm should need it. Is this missing? > > > IIUC, ptep_clear_flush() flushes tlb entry. So I think we are doing > unbatched flushing. Possibly a nice performance improvement later on > would be to try doing it batched. Suren can throw more light on it. Oh yeah.. thanks. > > One thing I was wondering is don't we need cache flush for the src > pages? mremap's move_page_tables() does it. IMHO, it's required here > as well. I commented in my reply, I also think it's needed. Otherwise for some arches I think we can have page containing stall data if not fully flushed before the movement. x86 is probably fine, though. > > > > > > > > > > Note: that (with only reading the documentation) it also kept me wondering > > > how the MMs are even implied from > > > > > > struct uffdio_move { > > > __u64 dst; /* Destination of move */ > > > __u64 src; /* Source of move */ > > > __u64 len; /* Number of bytes to move */ > > > __u64 mode; /* Flags controlling behavior of move */ > > > __s64 move; /* Number of bytes moved, or negated error */ > > > }; > > > > > > That probably has to be documented as well, in which address space dst and > > > src reside. > > > > Agreed, some better documentation will never hurt. Dst should be in the mm > > address space that was bound to the userfault descriptor. Src should be in > > the current mm address space. > > > > Thanks, > > > > -- > > Peter Xu > > >
[...] >>> Actually, even though I have no solid clue, but I had a feeling that there >>> can be some interesting way to leverage this across-mm movement, while >>> keeping things all safe (by e.g. elaborately requiring other proc to create >>> uffd and deliver to this proc). >> >> Okay, but no real use cases yet. > > I can provide a "not solid" example. I didn't mention it because it's > really something that just popped into my mind when thinking cross-mm, so I > never discussed with anyone yet nor shared it anywhere. > > Consider VM live upgrade in a generic form (e.g., no VFIO), we can do that > very efficiently with shmem or hugetlbfs, but not yet anonymous. We can do > extremely efficient postcopy live upgrade now with anonymous if with REMAP. > > Basically I see it a potential way of moving memory efficiently especially > with thp. It's an interesting use case indeed. The questions would be if this is (a) a use case we want to support; (b) why we need to make that decision now and add that feature. One question is if this kind of "moving memory between processes" really should be done, because intuitively SHMEM smells like the right thing to use here (two processes wanting to access the same memory). The downsides of shmem are lack of the shared zeropage and KSM. The shared zeropage is usually less of a concern for VMs, but KSM is. However, KSM will also disallow moving pages here. But all non-deduplicated ones could be moved. [I wondered whether moving KSM pages (rmap items) could be done; probably in some limited form with some more added complexity] > >> >>> >>> Considering Andrea's original version already contains those bits and all >>> above, I'd vote that we go ahead with supporting two MMs. >> >> You can do nasty things with that, as it stands, on the upstream codebase. >> >> If you pin the page in src_mm and move it to dst_mm, you successfully broke >> an invariant that "exclusive" means "no other references from other >> processes". That page is marked exclusive but it is, in fact, not exclusive. > > It is still exclusive to the dst mm? I see your point, but I think you're > taking exclusiveness altogether with pinning, and IMHO that may not be > always necessary? That's the definition of PAE. See do_wp_page() on when we reset PAE: when there are no other references, which implies no other references from other processes. Maybe you have "currently exclusively mapped" in mind, which is what the mapcount can be used for. > >> >> Once you achieved that, you can easily have src_mm not have MMF_HAS_PINNED, > > (I suppose you meant dst_mm here) Yes. > >> so you can just COW-share that page. Now you successfully broke the >> invariant that COW-shared pages must not be pinned. And you can even trigger >> VM_BUG_ONs, like in sanity_check_pinned_pages(). > > Yeah, that's really unfortunate. But frankly, I don't think it's the fault > of this new feature, but the rest. > > Let's imagine if the MMF_HAS_PINNED wasn't proposed as a per-mm flag, but > per-vma, which I don't see why we can't because it's simply a hint so far. > Then if we apply the same rule here, UFFDIO_REMAP won't even work for > single-mm as long as cross-vma. Then UFFDIO_REMAP as a whole feature will > be NACKed simply because of this.. Because of gup-fast we likely won't see that happening. And if we would, it could be handled (src_mm has the flag set, set it on the destination if the page maybe pinned after hiding it from gup-fast; or simply always copy the flag if set on the src). > > And I don't think anyone can guarantee a per-vma MMF_HAS_PINNED can never > happen, or any further change to pinning solution that may affect this. So > far it just looks unsafe to remap a pin page to me. It may be questionable to allow remapping pinned pages. > > I don't have a good suggestion here if this is a risk.. I'd think it risky > then to do REMAP over pinned pages no matter cross-mm or single-mm. It > means probably we just rule them out: folio_maybe_dma_pinned() may not even > be enough to be safe with fast-gup. We may need page_needs_cow_for_dma() > with proper write_protect_seq no matter cross-mm or single-mm? If you unmap and sync against GUP-fast, you can check after unmapping and remap and fail if it maybe pinned afterwards. Plus an early check upfront. > >> >> Can it all be fixed? Sure, with more complexity. For something without clear >> motivation, I'll have to pass. > > I think what you raised is a valid concern, but IMHO it's better fixed no > matter cross-mm or single-mm. What do you think? single-mm should at least not cause harm, but the semantics are questionable. cross-mm could, especially with malicious user space that wants to find ways of harming the kernel. I'll note that mremap with pinned pages works. > > In general, pinning lose its whole point here to me for an userspace either > if it DONTNEEDs it or REMAP it. What would be great to do here is we unpin > it upon DONTNEED/REMAP/whatever drops the page, because it loses its > coherency anyway, IMHO. Further, moving a part of a THP would fail either way, because the pinned THP cannot get split.
David, On Mon, Oct 16, 2023 at 08:01:10PM +0200, David Hildenbrand wrote: > [...] > > > > > Actually, even though I have no solid clue, but I had a feeling that there > > > > can be some interesting way to leverage this across-mm movement, while > > > > keeping things all safe (by e.g. elaborately requiring other proc to create > > > > uffd and deliver to this proc). > > > > > > Okay, but no real use cases yet. > > > > I can provide a "not solid" example. I didn't mention it because it's > > really something that just popped into my mind when thinking cross-mm, so I > > never discussed with anyone yet nor shared it anywhere. > > > > Consider VM live upgrade in a generic form (e.g., no VFIO), we can do that > > very efficiently with shmem or hugetlbfs, but not yet anonymous. We can do > > extremely efficient postcopy live upgrade now with anonymous if with REMAP. > > > > Basically I see it a potential way of moving memory efficiently especially > > with thp. > > It's an interesting use case indeed. The questions would be if this is (a) a > use case we want to support; (b) why we need to make that decision now and > add that feature. I would like to support that if nothing stops it from happening, but that's what we're discussing though.. For (b), I wanted to avoid UFFD_FEATURE_MOVE_CROSS_MM feature flag just for this, if they're already so close, not to mention current code already contains cross-mm support. If to support that live upgrade use case, I'd probably need to rework tlb flushing too to do the batching (actually tlb flush is not even needed for upgrade scenario..). I'm not sure whether Lokesh's use case would move large chunks, it'll be perfect if Suren did it altogether. But that one is much easier if transparent to userapps. Cross-mm is not transparent and need another feature knob, which I want to avoid if possible. > > One question is if this kind of "moving memory between processes" really > should be done, because intuitively SHMEM smells like the right thing to use > here (two processes wanting to access the same memory). That's the whole point, IMHO, where shmem cannot be used. As you said, on when someone cannot use file memory for some reason like ksm. > > The downsides of shmem are lack of the shared zeropage and KSM. The shared > zeropage is usually less of a concern for VMs, but KSM is. However, KSM will > also disallow moving pages here. But all non-deduplicated ones could be > moved. > > [I wondered whether moving KSM pages (rmap items) could be done; probably in > some limited form with some more added complexity] Yeah we can leave that complexity for later when really needed. Here cross-mm support, OTOH, isn't making it so complicated, IMHO. Btw, we don't even necessarily need to be able to migrate KSM pages for a VM live upgrade use case: we can unmerge the pages, upgrade, and wait for KSM to scan & merge again on the new binary / mmap. Userspace can have that control easily, afaiu, via existing madvise(). > > > > > > > > > > > > > > Considering Andrea's original version already contains those bits and all > > > > above, I'd vote that we go ahead with supporting two MMs. > > > > > > You can do nasty things with that, as it stands, on the upstream codebase. > > > > > > If you pin the page in src_mm and move it to dst_mm, you successfully broke > > > an invariant that "exclusive" means "no other references from other > > > processes". That page is marked exclusive but it is, in fact, not exclusive. > > > > It is still exclusive to the dst mm? I see your point, but I think you're > > taking exclusiveness altogether with pinning, and IMHO that may not be > > always necessary? > > That's the definition of PAE. See do_wp_page() on when we reset PAE: when > there are no other references, which implies no other references from other > processes. Maybe you have "currently exclusively mapped" in mind, which is > what the mapcount can be used for. Okay. > > > > > > > > > Once you achieved that, you can easily have src_mm not have MMF_HAS_PINNED, > > > > (I suppose you meant dst_mm here) > > Yes. > > > > > > so you can just COW-share that page. Now you successfully broke the > > > invariant that COW-shared pages must not be pinned. And you can even trigger > > > VM_BUG_ONs, like in sanity_check_pinned_pages(). > > > > Yeah, that's really unfortunate. But frankly, I don't think it's the fault > > of this new feature, but the rest. > > > > Let's imagine if the MMF_HAS_PINNED wasn't proposed as a per-mm flag, but > > per-vma, which I don't see why we can't because it's simply a hint so far. > > Then if we apply the same rule here, UFFDIO_REMAP won't even work for > > single-mm as long as cross-vma. Then UFFDIO_REMAP as a whole feature will > > be NACKed simply because of this.. > > Because of gup-fast we likely won't see that happening. And if we would, it > could be handled (src_mm has the flag set, set it on the destination if the > page maybe pinned after hiding it from gup-fast; or simply always copy the > flag if set on the src). > > > > > And I don't think anyone can guarantee a per-vma MMF_HAS_PINNED can never > > happen, or any further change to pinning solution that may affect this. So > > far it just looks unsafe to remap a pin page to me. > > It may be questionable to allow remapping pinned pages. > > > > > I don't have a good suggestion here if this is a risk.. I'd think it risky > > then to do REMAP over pinned pages no matter cross-mm or single-mm. It > > means probably we just rule them out: folio_maybe_dma_pinned() may not even > > be enough to be safe with fast-gup. We may need page_needs_cow_for_dma() > > with proper write_protect_seq no matter cross-mm or single-mm? > > If you unmap and sync against GUP-fast, you can check after unmapping and > remap and fail if it maybe pinned afterwards. Plus an early check upfront. > > > > > > > > > Can it all be fixed? Sure, with more complexity. For something without clear > > > motivation, I'll have to pass. > > > > I think what you raised is a valid concern, but IMHO it's better fixed no > > matter cross-mm or single-mm. What do you think? > > single-mm should at least not cause harm, but the semantics are > questionable. cross-mm could, especially with malicious user space that > wants to find ways of harming the kernel. For kernel, I think we're discussing to see whether it's safe to do so from kernel pov, e.g., whether to exclude pinned pages is part of that. For the user app, the dest process has provided the uffd descriptor on its own willingness, or be a child of the UFFDIO_MOVE issuer when used with EVENT_FORK, I assume that's already some form of safety check because it cannot be any process, but ones that are proactively in close cooperative with the issuer process. > > I'll note that mremap with pinned pages works. But that's not "by design", am I right? IOW, do we have any real pin user that rely on mremap() allowing pages to be moved? I don't see any word guarantee at least from man page that mremap() will make sure the PFN won't change after the movement.. even though it seems to be what happening now. Neither do I think when designing MMF_HAS_PINNED we kept that in mind that it won't be affected by someone mremap() pinned pages and we want to keep it working.. All of it just seems to be an accident.. One step back, we're free to define UFFDIO_MOVE anyway, and we don't necessarily need to always follow mremap(). E.g., mremap() also supports ksm pages, but IIUC we already decide to not support that for now on UFFDIO_MOVE. UFFDIO_MOVE seems all fine to make it clear on failing at pinned pages from the 1st day if that satisfies our goals, too. > > > > > In general, pinning lose its whole point here to me for an userspace either > > if it DONTNEEDs it or REMAP it. What would be great to do here is we unpin > > it upon DONTNEED/REMAP/whatever drops the page, because it loses its > > coherency anyway, IMHO. > > Further, moving a part of a THP would fail either way, because the pinned > THP cannot get split. > > -- > Cheers, > > David / dhildenb > Thanks,
On 16.10.23 21:01, Peter Xu wrote: > David, Hi Peter, >>> Basically I see it a potential way of moving memory efficiently especially >>> with thp. >> >> It's an interesting use case indeed. The questions would be if this is (a) a >> use case we want to support; (b) why we need to make that decision now and >> add that feature. > > I would like to support that if nothing stops it from happening, but that's > what we're discussing though.. > > For (b), I wanted to avoid UFFD_FEATURE_MOVE_CROSS_MM feature flag just for > this, if they're already so close, not to mention current code already > contains cross-mm support. Yeah, but that implementation is apparently not sufficiently correct yet. Don't get me wrong, but this feature is already complicated enough that we should really think twice if we want to make this even more complicated and harder to maintain -- because once it's in we all know it's hard to remove and we can easily end up with a maintenance nightmare without sufficiently good use cases. > > If to support that live upgrade use case, I'd probably need to rework tlb > flushing too to do the batching (actually tlb flush is not even needed for > upgrade scenario..). I'm not sure whether Lokesh's use case would move > large chunks, it'll be perfect if Suren did it altogether. But that one is > much easier if transparent to userapps. Cross-mm is not transparent and > need another feature knob, which I want to avoid if possible. And for me it's the other way around: the kernel doesn't have to support each and every use case. So we better think twice before we do something we can no longer undo easily. Further, as we see, this feature without cross-mm capabilities is perfectly usable for other use cases. So even limited initialy support is extremely valuable on its own. > >> >> One question is if this kind of "moving memory between processes" really >> should be done, because intuitively SHMEM smells like the right thing to use >> here (two processes wanting to access the same memory). > > That's the whole point, IMHO, where shmem cannot be used. As you said, on > when someone cannot use file memory for some reason like ksm. Right, but as I explore below KSM will at least prohibit remapping the KSM pages, taking some of the benefit away again. > >> >> The downsides of shmem are lack of the shared zeropage and KSM. The shared >> zeropage is usually less of a concern for VMs, but KSM is. However, KSM will >> also disallow moving pages here. But all non-deduplicated ones could be >> moved. >> >> [I wondered whether moving KSM pages (rmap items) could be done; probably in >> some limited form with some more added complexity] > > Yeah we can leave that complexity for later when really needed. Here > cross-mm support, OTOH, isn't making it so complicated, IMHO. > > Btw, we don't even necessarily need to be able to migrate KSM pages for a > VM live upgrade use case: we can unmerge the pages, upgrade, and wait for > KSM to scan & merge again on the new binary / mmap. Userspace can have > that control easily, afaiu, via existing madvise(). MADV_POPULATE_WRITE would do, yes. BTW, wasn't there a way to do VM live-upgrade using fork() and replacing the binary? I recall that there was at some time either an implementation in QEMU or a proposal for an implementation; but I don't know how VM memory was provided. It's certainly harder to move VM memory using fork(). [...] >> >> single-mm should at least not cause harm, but the semantics are >> questionable. cross-mm could, especially with malicious user space that >> wants to find ways of harming the kernel. > > For kernel, I think we're discussing to see whether it's safe to do so from > kernel pov, e.g., whether to exclude pinned pages is part of that. > > For the user app, the dest process has provided the uffd descriptor on its > own willingness, or be a child of the UFFDIO_MOVE issuer when used with > EVENT_FORK, I assume that's already some form of safety check because it > cannot be any process, but ones that are proactively in close cooperative > with the issuer process. Is that and will that remain the case? I know people have been working on transparent user-space swapping using monitor processes using uffd. I thought there would have been ways to achieve that without any corporation of the dst. > >> >> I'll note that mremap with pinned pages works. > > But that's not "by design", am I right? IOW, do we have any real pin user > that rely on mremap() allowing pages to be moved? If in doubt, usually "probably yes". Hard to tell if the "remap" in mremap indicates that we are simply remapping pages , and not moving data. > > I don't see any word guarantee at least from man page that mremap() will > make sure the PFN won't change after the movement.. even though it seems to > be what happening now. If you managed to support page pinning with migration the exact PFN doesn't matter. Likely nobody would implement that due to the tracking complexity. > > Neither do I think when designing MMF_HAS_PINNED we kept that in mind that > it won't be affected by someone mremap() pinned pages and we want to keep > it working.. > > All of it just seems to be an accident.. Not necessarily my opinion, but doesn't really matter here. It's working. At least It's reasonable to have part of a THP pinned and mremapping the other part of the THP. That makes things more tricky ... because you only know that the THP is pinned. > > One step back, we're free to define UFFDIO_MOVE anyway, and we don't > necessarily need to always follow mremap(). E.g., mremap() also supports > ksm pages, but IIUC we already decide to not support that for now on > UFFDIO_MOVE. UFFDIO_MOVE seems all fine to make it clear on failing at > pinned pages from the 1st day if that satisfies our goals, too. Yes.
David, On Tue, Oct 17, 2023 at 05:55:10PM +0200, David Hildenbrand wrote: > Don't get me wrong, but this feature is already complicated enough that we > should really think twice if we want to make this even more complicated and > harder to maintain -- because once it's in we all know it's hard to remove > and we can easily end up with a maintenance nightmare without sufficiently > good use cases. Yes I agree it's non-trivial. My point is adding cross-mm doesn't make it even more complicated.. afaics. For example, could you provide a list of things that will be different to support single mm or cross mm? I see two things that can be different, but I'd rather have all of them even if single-mm.. - cgroup: I assume single-mm may avoid uncharge and charge again, but I prefer it be there even if we only allow single-mm. For example, I'm not 100% sure whether memcg won't start to behave differently according to vma attribute in the future. - page pinning: I assume for single-mm we can avoid checking page pinning based on the fact that MMF_HAS_PINNED is per-mm, but I also prefer we fail explicitly on pinned pages over UFFDIO_MOVE because it doesn't sound correct, and avoid future changes on top of pinning solution that can change the assumption that "move a pin page within mm" is ok. Is there anything else that will be different? Did I miss something important? [...] > BTW, wasn't there a way to do VM live-upgrade using fork() and replacing the > binary? I recall that there was at some time either an implementation in > QEMU or a proposal for an implementation; but I don't know how VM memory was > provided. It's certainly harder to move VM memory using fork(). Maybe you meant the cpr project. I didn't actually follow that much previously (and will need to follow more after I took the migration duties.. when there's a new post), but IIUC at least the latest version needs to go with file memory only, not anonymous: https://lore.kernel.org/all/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/ Guest RAM must be non-volatile across reboot, which can be achieved by backing it with a dax device, or /dev/shm PKRAM as proposed in... Guest RAM must be backed by a memory backend with share=on, but cannot be memory-backend-ram. The memory is re-mmap'd in the updated process, so guest ram is efficiently preserved in place My understanding is there used to have solution for anonymous but that needs extra kernel changes (MADV_DOEXEC). https://lore.kernel.org/linux-mm/1595869887-23307-1-git-send-email-anthony.yznaga@oracle.com/ I saw that you were part of the discussion, so maybe you will remember some more clue of that part. IIUC one core requirement of the whole approach is also that it will cover VFIO and maintenance of device DMA mappings, in which case it'll be different with any approach to leverage UFFDIO_MOVE because VFIO will not be allowed here; again I hope we start with forbid pinning. But it should be much cleaner on the design when with UFFDIO_MOVE, just not working with VFIO. One thing I'd need to measure is latency of UFFDIO_MOVE on page fault resolutions. I expect no more than tens of microseconds or even less. Should be drastically smaller than remote postcopy anyway. I'm probably off topic.. To go back: let's try to figure out what is special with cross-mm support. It'll be very weird in the future for anyone to propose a patch just add a feature flag and declaring cross-mm support, if the code is mostly all there. Nothing stops us from discussing what a cross-mm design will need. [...] > Is that and will that remain the case? I know people have been working on > transparent user-space swapping using monitor processes using uffd. I > thought there would have been ways to achieve that without any corporation > of the dst. Any example? For what I am aware, all corporation requires uffd desc forwarding. I think the trick here is any userfaultfd desc must be created by its own process, so far nobody else. That's more or less saying "I want to do this" from its own opinion. The next is forwarding that to someone else. Parent process is fine taking uffd of child with EVENT_FORK, as I mentioned, but besides that nothing else I can think of that can violate this guard to manipulate a random process. Thanks,
Hi Suren,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on next-20231017]
[cannot apply to linus/master v6.6-rc6]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Suren-Baghdasaryan/mm-rmap-support-move-to-different-root-anon_vma-in-folio_move_anon_rmap/20231009-144552
base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link: https://lore.kernel.org/r/20231009064230.2952396-3-surenb%40google.com
patch subject: [PATCH v3 2/3] userfaultfd: UFFDIO_MOVE uABI
config: i386-randconfig-141-20231017 (https://download.01.org/0day-ci/archive/20231018/202310180338.zTpcYECK-lkp@intel.com/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce: (https://download.01.org/0day-ci/archive/20231018/202310180338.zTpcYECK-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202310180338.zTpcYECK-lkp@intel.com/
smatch warnings:
mm/userfaultfd.c:1380 remap_pages() warn: unsigned 'src_start + len - src_addr' is never less than zero.
vim +1380 mm/userfaultfd.c
1195
1196 /**
1197 * remap_pages - remap arbitrary anonymous pages of an existing vma
1198 * @dst_start: start of the destination virtual memory range
1199 * @src_start: start of the source virtual memory range
1200 * @len: length of the virtual memory range
1201 *
1202 * remap_pages() remaps arbitrary anonymous pages atomically in zero
1203 * copy. It only works on non shared anonymous pages because those can
1204 * be relocated without generating non linear anon_vmas in the rmap
1205 * code.
1206 *
1207 * It provides a zero copy mechanism to handle userspace page faults.
1208 * The source vma pages should have mapcount == 1, which can be
1209 * enforced by using madvise(MADV_DONTFORK) on src vma.
1210 *
1211 * The thread receiving the page during the userland page fault
1212 * will receive the faulting page in the source vma through the network,
1213 * storage or any other I/O device (MADV_DONTFORK in the source vma
1214 * avoids remap_pages() to fail with -EBUSY if the process forks before
1215 * remap_pages() is called), then it will call remap_pages() to map the
1216 * page in the faulting address in the destination vma.
1217 *
1218 * This userfaultfd command works purely via pagetables, so it's the
1219 * most efficient way to move physical non shared anonymous pages
1220 * across different virtual addresses. Unlike mremap()/mmap()/munmap()
1221 * it does not create any new vmas. The mapping in the destination
1222 * address is atomic.
1223 *
1224 * It only works if the vma protection bits are identical from the
1225 * source and destination vma.
1226 *
1227 * It can remap non shared anonymous pages within the same vma too.
1228 *
1229 * If the source virtual memory range has any unmapped holes, or if
1230 * the destination virtual memory range is not a whole unmapped hole,
1231 * remap_pages() will fail respectively with -ENOENT or -EEXIST. This
1232 * provides a very strict behavior to avoid any chance of memory
1233 * corruption going unnoticed if there are userland race conditions.
1234 * Only one thread should resolve the userland page fault at any given
1235 * time for any given faulting address. This means that if two threads
1236 * try to both call remap_pages() on the same destination address at the
1237 * same time, the second thread will get an explicit error from this
1238 * command.
1239 *
1240 * The command retval will return "len" is successful. The command
1241 * however can be interrupted by fatal signals or errors. If
1242 * interrupted it will return the number of bytes successfully
1243 * remapped before the interruption if any, or the negative error if
1244 * none. It will never return zero. Either it will return an error or
1245 * an amount of bytes successfully moved. If the retval reports a
1246 * "short" remap, the remap_pages() command should be repeated by
1247 * userland with src+retval, dst+reval, len-retval if it wants to know
1248 * about the error that interrupted it.
1249 *
1250 * The UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES flag can be specified to
1251 * prevent -ENOENT errors to materialize if there are holes in the
1252 * source virtual range that is being remapped. The holes will be
1253 * accounted as successfully remapped in the retval of the
1254 * command. This is mostly useful to remap hugepage naturally aligned
1255 * virtual regions without knowing if there are transparent hugepage
1256 * in the regions or not, but preventing the risk of having to split
1257 * the hugepmd during the remap.
1258 *
1259 * If there's any rmap walk that is taking the anon_vma locks without
1260 * first obtaining the folio lock (the only current instance is
1261 * folio_referenced), they will have to verify if the folio->mapping
1262 * has changed after taking the anon_vma lock. If it changed they
1263 * should release the lock and retry obtaining a new anon_vma, because
1264 * it means the anon_vma was changed by remap_pages() before the lock
1265 * could be obtained. This is the only additional complexity added to
1266 * the rmap code to provide this anonymous page remapping functionality.
1267 */
1268 ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm,
1269 unsigned long dst_start, unsigned long src_start,
1270 unsigned long len, __u64 mode)
1271 {
1272 struct vm_area_struct *src_vma, *dst_vma;
1273 unsigned long src_addr, dst_addr;
1274 pmd_t *src_pmd, *dst_pmd;
1275 long err = -EINVAL;
1276 ssize_t moved = 0;
1277
1278 /*
1279 * Sanitize the command parameters:
1280 */
1281 BUG_ON(src_start & ~PAGE_MASK);
1282 BUG_ON(dst_start & ~PAGE_MASK);
1283 BUG_ON(len & ~PAGE_MASK);
1284
1285 /* Does the address range wrap, or is the span zero-sized? */
1286 BUG_ON(src_start + len <= src_start);
1287 BUG_ON(dst_start + len <= dst_start);
1288
1289 /*
1290 * Because these are read sempahores there's no risk of lock
1291 * inversion.
1292 */
1293 mmap_read_lock(dst_mm);
1294 if (dst_mm != src_mm)
1295 mmap_read_lock(src_mm);
1296
1297 /*
1298 * Make sure the vma is not shared, that the src and dst remap
1299 * ranges are both valid and fully within a single existing
1300 * vma.
1301 */
1302 src_vma = find_vma(src_mm, src_start);
1303 if (!src_vma || (src_vma->vm_flags & VM_SHARED))
1304 goto out;
1305 if (src_start < src_vma->vm_start ||
1306 src_start + len > src_vma->vm_end)
1307 goto out;
1308
1309 dst_vma = find_vma(dst_mm, dst_start);
1310 if (!dst_vma || (dst_vma->vm_flags & VM_SHARED))
1311 goto out;
1312 if (dst_start < dst_vma->vm_start ||
1313 dst_start + len > dst_vma->vm_end)
1314 goto out;
1315
1316 err = validate_remap_areas(src_vma, dst_vma);
1317 if (err)
1318 goto out;
1319
1320 for (src_addr = src_start, dst_addr = dst_start;
1321 src_addr < src_start + len;) {
1322 spinlock_t *ptl;
1323 pmd_t dst_pmdval;
1324 unsigned long step_size;
1325
1326 BUG_ON(dst_addr >= dst_start + len);
1327 /*
1328 * Below works because anonymous area would not have a
1329 * transparent huge PUD. If file-backed support is added,
1330 * that case would need to be handled here.
1331 */
1332 src_pmd = mm_find_pmd(src_mm, src_addr);
1333 if (unlikely(!src_pmd)) {
1334 if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) {
1335 err = -ENOENT;
1336 break;
1337 }
1338 src_pmd = mm_alloc_pmd(src_mm, src_addr);
1339 if (unlikely(!src_pmd)) {
1340 err = -ENOMEM;
1341 break;
1342 }
1343 }
1344 dst_pmd = mm_alloc_pmd(dst_mm, dst_addr);
1345 if (unlikely(!dst_pmd)) {
1346 err = -ENOMEM;
1347 break;
1348 }
1349
1350 dst_pmdval = pmdp_get_lockless(dst_pmd);
1351 /*
1352 * If the dst_pmd is mapped as THP don't override it and just
1353 * be strict. If dst_pmd changes into TPH after this check, the
1354 * remap_pages_huge_pmd() will detect the change and retry
1355 * while remap_pages_pte() will detect the change and fail.
1356 */
1357 if (unlikely(pmd_trans_huge(dst_pmdval))) {
1358 err = -EEXIST;
1359 break;
1360 }
1361
1362 ptl = pmd_trans_huge_lock(src_pmd, src_vma);
1363 if (ptl) {
1364 if (pmd_devmap(*src_pmd)) {
1365 spin_unlock(ptl);
1366 err = -ENOENT;
1367 break;
1368 }
1369
1370 /*
1371 * Check if we can move the pmd without
1372 * splitting it. First check the address
1373 * alignment to be the same in src/dst. These
1374 * checks don't actually need the PT lock but
1375 * it's good to do it here to optimize this
1376 * block away at build time if
1377 * CONFIG_TRANSPARENT_HUGEPAGE is not set.
1378 */
1379 if ((src_addr & ~HPAGE_PMD_MASK) || (dst_addr & ~HPAGE_PMD_MASK) ||
> 1380 src_start + len - src_addr < HPAGE_PMD_SIZE || !pmd_none(dst_pmdval)) {
On 17.10.23 20:59, Peter Xu wrote: > David, > > On Tue, Oct 17, 2023 at 05:55:10PM +0200, David Hildenbrand wrote: >> Don't get me wrong, but this feature is already complicated enough that we >> should really think twice if we want to make this even more complicated and >> harder to maintain -- because once it's in we all know it's hard to remove >> and we can easily end up with a maintenance nightmare without sufficiently >> good use cases. > > Yes I agree it's non-trivial. My point is adding cross-mm doesn't make it > even more complicated.. afaics. That's not my main point. It can easily become a maintenance burden without any real use cases yet that we are willing to support. > > For example, could you provide a list of things that will be different to > support single mm or cross mm? I see two things that can be different, but > I'd rather have all of them even if single-mm.. > > - cgroup: I assume single-mm may avoid uncharge and charge again, but I > prefer it be there even if we only allow single-mm. For example, I'm > not 100% sure whether memcg won't start to behave differently according > to vma attribute in the future. > > - page pinning: I assume for single-mm we can avoid checking page pinning > based on the fact that MMF_HAS_PINNED is per-mm, but I also prefer we > fail explicitly on pinned pages over UFFDIO_MOVE because it doesn't > sound correct, and avoid future changes on top of pinning solution that > can change the assumption that "move a pin page within mm" is ok. > > Is there anything else that will be different? Did I miss something > important? Again, that's not my main point. All I'm asking for is to separate it out, make it a separate flag, and include it once we have reasonable use cases that we are actually willing to support -- including actual data why it's beneficial to have. For the single-mm use it has been shown that there are reasonable, existing use cases exist, and I think we are willing to support that. This patch set is close to doubling (!) the size of mm/userfaultfd.c, and it already has every possible smell of maintanance nightmare IMHO. It does things that shouldn't be specific to some MM subsystem. I'm happy to see any possible complexity reduced. Moving pages between MMs is added complexity. But I will stop arguing further; I hope I made my point clear and I have other things to work on than fighting against overly-complicated uffd features. > > [...] > >> BTW, wasn't there a way to do VM live-upgrade using fork() and replacing the >> binary? I recall that there was at some time either an implementation in >> QEMU or a proposal for an implementation; but I don't know how VM memory was >> provided. It's certainly harder to move VM memory using fork(). > > Maybe you meant the cpr project. I didn't actually follow that much > previously (and will need to follow more after I took the migration > duties.. when there's a new post), but IIUC at least the latest version > needs to go with file memory only, not anonymous: > > https://lore.kernel.org/all/1658851843-236870-1-git-send-email-steven.sistare@oracle.com/ > > Guest RAM must be non-volatile across reboot, which can be achieved by > backing it with a dax device, or /dev/shm PKRAM as proposed in... > > Guest RAM must be backed by a memory backend with share=on, but > cannot be memory-backend-ram. The memory is re-mmap'd in the > updated process, so guest ram is efficiently preserved in place > > My understanding is there used to have solution for anonymous but that > needs extra kernel changes (MADV_DOEXEC). Probably, I also stumbled over a paper from 2019 that mentioned that that. > > https://lore.kernel.org/linux-mm/1595869887-23307-1-git-send-email-anthony.yznaga@oracle.com/ > > I saw that you were part of the discussion, so maybe you will remember some > more clue of that part. > Ouch, 2020. But my comments were only regarding mshare, not MADV_DOEXEC. In fact, I don't even know why both discussions/threads show up as a single one there.. > IIUC one core requirement of the whole approach is also that it will cover > VFIO and maintenance of device DMA mappings, in which case it'll be > different with any approach to leverage UFFDIO_MOVE because VFIO will not > be allowed here; again I hope we start with forbid pinning. But it should > be much cleaner on the design when with UFFDIO_MOVE, just not working with > VFIO. > > One thing I'd need to measure is latency of UFFDIO_MOVE on page fault > resolutions. I expect no more than tens of microseconds or even less. > Should be drastically smaller than remote postcopy anyway. > > I'm probably off topic.. To go back: let's try to figure out what is > special with cross-mm support. It'll be very weird in the future for > anyone to propose a patch just add a feature flag and declaring cross-mm > support, if the code is mostly all there. Nothing stops us from discussing > what a cross-mm design will need. Again, I hope I made my point clear. > > [...] > >> Is that and will that remain the case? I know people have been working on >> transparent user-space swapping using monitor processes using uffd. I >> thought there would have been ways to achieve that without any corporation >> of the dst. > > Any example? Nothing concrete, I only heard about uffd monitors that implement user-space based swapping. I don't recall if they require some kind of support from a library that gets loaded into these processes, Same thoughts regarding CRIU using uffd. > > For what I am aware, all corporation requires uffd desc forwarding. I > think the trick here is any userfaultfd desc must be created by its own > process, so far nobody else. That's more or less saying "I want to do > this" from its own opinion. The next is forwarding that to someone else. > Parent process is fine taking uffd of child with EVENT_FORK, as I > mentioned, but besides that nothing else I can think of that can violate > this guard to manipulate a random process. Do you have any idea how CRIU makes that work (at least I recall that they wanted to use UFFD).
On Thu, Oct 19, 2023 at 05:41:01PM +0200, David Hildenbrand wrote: > That's not my main point. It can easily become a maintenance burden without > any real use cases yet that we are willing to support. That's why I requested a few times that we can discuss the complexity of cross-mm support already here, and I'm all ears if I missed something on the "maintenance burden" part.. I started by listing what I think might be different, and we can easily speedup single-mm with things like "if (ctx->mm != mm)" checks with e.g. memcg, just like what this patch already did with pgtable depositions. We keep saying "maintenance burden" but we refuse to discuss what is that.. I'll leave that to Suren and Lokesh to decide. For me the worst case is one more flag which might be confusing, which is not the end of the world.. Suren, you may need to work more thoroughly to remove cross-mm implications if so, just like when renaming REMAP to MOVE. Thanks,
On Thu, Oct 19, 2023 at 12:53 PM Peter Xu <peterx@redhat.com> wrote: > > On Thu, Oct 19, 2023 at 05:41:01PM +0200, David Hildenbrand wrote: > > That's not my main point. It can easily become a maintenance burden without > > any real use cases yet that we are willing to support. > > That's why I requested a few times that we can discuss the complexity of > cross-mm support already here, and I'm all ears if I missed something on > the "maintenance burden" part.. > > I started by listing what I think might be different, and we can easily > speedup single-mm with things like "if (ctx->mm != mm)" checks with > e.g. memcg, just like what this patch already did with pgtable depositions. > > We keep saying "maintenance burden" but we refuse to discuss what is that.. > > I'll leave that to Suren and Lokesh to decide. For me the worst case is > one more flag which might be confusing, which is not the end of the world.. > Suren, you may need to work more thoroughly to remove cross-mm implications > if so, just like when renaming REMAP to MOVE. Hi Folks, Sorry, I'm just catching up on all the comments in this thread after a week-long absence. Will be addressing other questions separately but for cross-mm one, I think the best way forward would be for me to split this patch into two with the second one adding cross-mm support. That will clearly show how much additional code that requires and will make it easier for us to decide whether to support it or not. TBH, I don't see the need for an additional flag even if the initial version will be merged without cross-mm support. Once it's added the manpage can mention that starting with a specific Linux version cross-mm is supported, no? Also from my quick read, it sounds like we want to prevent movements of pinned pages regardless of cross-mm support. Is my understanding correct? Thanks, Suren. > > Thanks, > > -- > Peter Xu >
On Thu, Oct 19, 2023 at 01:02:39PM -0700, Suren Baghdasaryan wrote: > Hi Folks, > Sorry, I'm just catching up on all the comments in this thread after a Not a problem. > week-long absence. Will be addressing other questions separately but > for cross-mm one, I think the best way forward would be for me to > split this patch into two with the second one adding cross-mm support. > That will clearly show how much additional code that requires and will > make it easier for us to decide whether to support it or not. Sounds good, thanks for that extra work. > TBH, I don't see the need for an additional flag even if the initial > version will be merged without cross-mm support. Once it's added the > manpage can mention that starting with a specific Linux version > cross-mm is supported, no? It's about how an user app knows what the kernel supports. On kernels that only support single-mm, UFFDIO_MOVE should fail if it found ctx->mm != current->mm. I think the best way to let the user app be clear of what happened is one new feature bit if cross-mm will be supported separately. Or the userapp will need to rely on a specific failure code of UFFDIO_MOVE, and only until the 1st MOVE being triggered. Not as clear, IMHO. > Also from my quick read, it sounds like we want to prevent movements > of pinned pages regardless of cross-mm support. Is my understanding > correct? I prefer that, but that's only my 2 cents. I just don't see how remap can work with pin. IIUC pin is about coherency of processor view and DMA view. Then if so the VA is the only identifier of a "page" for an user app because real pfn is hidden, and remap changes that VA. So it doesn't make sense to me to remap a pin in whatever form. For check pinning: I think I used to mention that it may again require proper locking over mm.write_protect_seq like fork() paths. No, when thinking again I think I was wrong.. write_protect_seq requires mmap write lock, definitely not good. We can do what David mentioned before, after ptep_clear_flush() (so pte is cleared) we recheck page pinning, if pinned fail MOVE and put the page back. Note that we can't do that check after installing it into dest pgtables, because then someone can start to pin it from dest mm already. Thanks,
On Thu, Oct 12, 2023 at 3:00 PM Peter Xu <peterx@redhat.com> wrote: > > On Sun, Oct 08, 2023 at 11:42:27PM -0700, Suren Baghdasaryan wrote: > > From: Andrea Arcangeli <aarcange@redhat.com> > > > > Implement the uABI of UFFDIO_MOVE ioctl. > > UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application > > needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are > > available (in userspace) for recycling, as is usually the case in heap > > compaction algorithms, then we can avoid the page allocation and memcpy > > (done by UFFDIO_COPY). Also, since the pages are recycled in the > > userspace, we avoid the need to release (via madvise) the pages back to > > the kernel [2]. > > We see over 40% reduction (on a Google pixel 6 device) in the compacting > > thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was > > measured using a benchmark that emulates a heap compaction implementation > > using userfaultfd (to allow concurrent accesses by application threads). > > More details of the usecase are explained in [2]. > > Furthermore, UFFDIO_MOVE enables moving swapped-out pages without > > touching them within the same vma. Today, it can only be done by mremap, > > however it forces splitting the vma. > > > > [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ > > [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ > > > > Update for the ioctl_userfaultfd(2) manpage: > > > > UFFDIO_MOVE > > (Since Linux xxx) Move a continuous memory chunk into the > > userfault registered range and optionally wake up the blocked > > thread. The source and destination addresses and the number of > > bytes to move are specified by the src, dst, and len fields of > > the uffdio_move structure pointed to by argp: > > > > struct uffdio_move { > > __u64 dst; /* Destination of move */ > > __u64 src; /* Source of move */ > > __u64 len; /* Number of bytes to move */ > > __u64 mode; /* Flags controlling behavior of move */ > > __s64 move; /* Number of bytes moved, or negated error */ > > }; > > > > The following value may be bitwise ORed in mode to change the > > behavior of the UFFDIO_MOVE operation: > > > > UFFDIO_MOVE_MODE_DONTWAKE > > Do not wake up the thread that waits for page-fault > > resolution > > > > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES > > Allow holes in the source virtual range that is being moved. > > When not specified, the holes will result in ENOENT error. > > When specified, the holes will be accounted as successfully > > moved memory. This is mostly useful to move hugepage aligned > > virtual regions without knowing if there are transparent > > hugepages in the regions or not, but preventing the risk of > > having to split the hugepage during the operation. > > > > The move field is used by the kernel to return the number of > > bytes that was actually moved, or an error (a negated errno- > > style value). If the value returned in move doesn't match the > > value that was specified in len, the operation fails with the > > error EAGAIN. The move field is output-only; it is not read by > > the UFFDIO_MOVE operation. > > > > The operation may fail for various reasons. Usually, remapping of > > pages that are not exclusive to the given process fail; once KSM > > might deduplicate pages or fork() COW-shares pages during fork() > > with child processes, they are no longer exclusive. Further, the > > kernel might only perform lightweight checks for detecting whether > > the pages are exclusive, and return -EBUSY in case that check fails. > > To make the operation more likely to succeed, KSM should be > > disabled, fork() should be avoided or MADV_DONTFORK should be > > configured for the source VMA before fork(). > > > > This ioctl(2) operation returns 0 on success. In this case, the > > entire area was moved. On error, -1 is returned and errno is > > set to indicate the error. Possible errors include: > > > > EAGAIN The number of bytes moved (i.e., the value returned in > > the move field) does not equal the value that was > > specified in the len field. > > > > EINVAL Either dst or len was not a multiple of the system page > > size, or the range specified by src and len or dst and len > > was invalid. > > > > EINVAL An invalid bit was specified in the mode field. > > > > ENOENT > > The source virtual memory range has unmapped holes and > > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. > > > > EEXIST > > The destination virtual memory range is fully or partially > > mapped. > > > > EBUSY > > The pages in the source virtual memory range are not > > exclusive to the process. The kernel might only perform > > lightweight checks for detecting whether the pages are > > exclusive. To make the operation more likely to succeed, > > KSM should be disabled, fork() should be avoided or > > MADV_DONTFORK should be configured for the source virtual > > memory area before fork(). > > > > ENOMEM Allocating memory needed for the operation failed. > > > > ESRCH > > The faulting process has exited at the time of a > > Nit pick comment for future man page: there's no faulting process in this > context. Perhaps "target process"? Ack. > > > UFFDIO_MOVE operation. > > > > Signed-off-by: Andrea Arcangeli <aarcange@redhat.com> > > Signed-off-by: Suren Baghdasaryan <surenb@google.com> > > --- > > Documentation/admin-guide/mm/userfaultfd.rst | 3 + > > fs/userfaultfd.c | 63 ++ > > include/linux/rmap.h | 5 + > > include/linux/userfaultfd_k.h | 12 + > > include/uapi/linux/userfaultfd.h | 29 +- > > mm/huge_memory.c | 138 +++++ > > mm/khugepaged.c | 3 + > > mm/rmap.c | 6 + > > mm/userfaultfd.c | 602 +++++++++++++++++++ > > 9 files changed, 860 insertions(+), 1 deletion(-) > > > > diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst > > index 203e26da5f92..e5cc8848dcb3 100644 > > --- a/Documentation/admin-guide/mm/userfaultfd.rst > > +++ b/Documentation/admin-guide/mm/userfaultfd.rst > > @@ -113,6 +113,9 @@ events, except page fault notifications, may be generated: > > areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the analogous feature indicating > > support for shmem virtual memory areas. > > > > +- ``UFFD_FEATURE_MOVE`` indicates that the kernel supports moving an > > + existing page contents from userspace. > > + > > The userland application should set the feature flags it intends to use > > when invoking the ``UFFDIO_API`` ioctl, to request that those features be > > enabled if supported. > > diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c > > index a7c6ef764e63..ac52e0f99a69 100644 > > --- a/fs/userfaultfd.c > > +++ b/fs/userfaultfd.c > > @@ -2039,6 +2039,66 @@ static inline unsigned int uffd_ctx_features(__u64 user_features) > > return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED; > > } > > > > +static int userfaultfd_remap(struct userfaultfd_ctx *ctx, > > + unsigned long arg) > > If we do want to rename from REMAP to MOVE, we'd better rename the > functions too, as "remap" still exists all over the place.. Ok. I thought that since the current implementation only remaps and never copies it would be correct to keep "remap" in these internal names and change that later if we support copying. But I'm fine with renaming them now to avoid confusion. Will do. > > > +{ > > + __s64 ret; > > + struct uffdio_move uffdio_move; > > + struct uffdio_move __user *user_uffdio_move; > > + struct userfaultfd_wake_range range; > > + > > + user_uffdio_move = (struct uffdio_move __user *) arg; > > + > > + ret = -EAGAIN; > > + if (atomic_read(&ctx->mmap_changing)) > > + goto out; > > I didn't notice this before, but I think we need to re-check this after > taking target mm's mmap read lock.. Ack. > > maybe we'd want to pass in ctx* into remap_pages(), more reasoning below. Makes sense. > > > + > > + ret = -EFAULT; > > + if (copy_from_user(&uffdio_move, user_uffdio_move, > > + /* don't copy "remap" last field */ > > s/remap/move/ Ack. > > > + sizeof(uffdio_move)-sizeof(__s64))) > > + goto out; > > + > > + ret = validate_range(ctx->mm, uffdio_move.dst, uffdio_move.len); > > + if (ret) > > + goto out; > > + > > + ret = validate_range(current->mm, uffdio_move.src, uffdio_move.len); > > + if (ret) > > + goto out; > > + > > + ret = -EINVAL; > > + if (uffdio_move.mode & ~(UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES| > > + UFFDIO_MOVE_MODE_DONTWAKE)) > > + goto out; > > + > > + if (mmget_not_zero(ctx->mm)) { > > + ret = remap_pages(ctx->mm, current->mm, > > + uffdio_move.dst, uffdio_move.src, > > + uffdio_move.len, uffdio_move.mode); > > + mmput(ctx->mm); > > + } else { > > + return -ESRCH; > > + } > > + > > + if (unlikely(put_user(ret, &user_uffdio_move->move))) > > + return -EFAULT; > > + if (ret < 0) > > + goto out; > > + > > + /* len == 0 would wake all */ > > + BUG_ON(!ret); > > + range.len = ret; > > + if (!(uffdio_move.mode & UFFDIO_MOVE_MODE_DONTWAKE)) { > > + range.start = uffdio_move.dst; > > + wake_userfault(ctx, &range); > > + } > > + ret = range.len == uffdio_move.len ? 0 : -EAGAIN; > > + > > +out: > > + return ret; > > +} > > + > > /* > > * userland asks for a certain API version and we return which bits > > * and ioctl commands are implemented in this kernel for such API > > @@ -2131,6 +2191,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd, > > case UFFDIO_ZEROPAGE: > > ret = userfaultfd_zeropage(ctx, arg); > > break; > > + case UFFDIO_MOVE: > > + ret = userfaultfd_remap(ctx, arg); > > + break; > > case UFFDIO_WRITEPROTECT: > > ret = userfaultfd_writeprotect(ctx, arg); > > break; > > diff --git a/include/linux/rmap.h b/include/linux/rmap.h > > index b26fe858fd44..8034eda972e5 100644 > > --- a/include/linux/rmap.h > > +++ b/include/linux/rmap.h > > @@ -121,6 +121,11 @@ static inline void anon_vma_lock_write(struct anon_vma *anon_vma) > > down_write(&anon_vma->root->rwsem); > > } > > > > +static inline int anon_vma_trylock_write(struct anon_vma *anon_vma) > > +{ > > + return down_write_trylock(&anon_vma->root->rwsem); > > +} > > + > > static inline void anon_vma_unlock_write(struct anon_vma *anon_vma) > > { > > up_write(&anon_vma->root->rwsem); > > diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h > > index f2dc19f40d05..ce8d20b57e8c 100644 > > --- a/include/linux/userfaultfd_k.h > > +++ b/include/linux/userfaultfd_k.h > > @@ -93,6 +93,18 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm, > > extern long uffd_wp_range(struct vm_area_struct *vma, > > unsigned long start, unsigned long len, bool enable_wp); > > > > +/* remap_pages */ > > +void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2); > > +void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2); > > +ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > + unsigned long dst_start, unsigned long src_start, > > + unsigned long len, __u64 flags); > > +int remap_pages_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > + pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, > > + struct vm_area_struct *dst_vma, > > + struct vm_area_struct *src_vma, > > + unsigned long dst_addr, unsigned long src_addr); > > + > > /* mm helpers */ > > static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma, > > struct vm_userfaultfd_ctx vm_ctx) > > diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h > > index 0dbc81015018..2841e4ea8f2c 100644 > > --- a/include/uapi/linux/userfaultfd.h > > +++ b/include/uapi/linux/userfaultfd.h > > @@ -41,7 +41,8 @@ > > UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \ > > UFFD_FEATURE_WP_UNPOPULATED | \ > > UFFD_FEATURE_POISON | \ > > - UFFD_FEATURE_WP_ASYNC) > > + UFFD_FEATURE_WP_ASYNC | \ > > + UFFD_FEATURE_MOVE) > > #define UFFD_API_IOCTLS \ > > ((__u64)1 << _UFFDIO_REGISTER | \ > > (__u64)1 << _UFFDIO_UNREGISTER | \ > > @@ -50,6 +51,7 @@ > > ((__u64)1 << _UFFDIO_WAKE | \ > > (__u64)1 << _UFFDIO_COPY | \ > > (__u64)1 << _UFFDIO_ZEROPAGE | \ > > + (__u64)1 << _UFFDIO_MOVE | \ > > (__u64)1 << _UFFDIO_WRITEPROTECT | \ > > (__u64)1 << _UFFDIO_CONTINUE | \ > > (__u64)1 << _UFFDIO_POISON) > > @@ -73,6 +75,7 @@ > > #define _UFFDIO_WAKE (0x02) > > #define _UFFDIO_COPY (0x03) > > #define _UFFDIO_ZEROPAGE (0x04) > > +#define _UFFDIO_MOVE (0x05) > > #define _UFFDIO_WRITEPROTECT (0x06) > > #define _UFFDIO_CONTINUE (0x07) > > #define _UFFDIO_POISON (0x08) > > @@ -92,6 +95,8 @@ > > struct uffdio_copy) > > #define UFFDIO_ZEROPAGE _IOWR(UFFDIO, _UFFDIO_ZEROPAGE, \ > > struct uffdio_zeropage) > > +#define UFFDIO_MOVE _IOWR(UFFDIO, _UFFDIO_MOVE, \ > > + struct uffdio_move) > > #define UFFDIO_WRITEPROTECT _IOWR(UFFDIO, _UFFDIO_WRITEPROTECT, \ > > struct uffdio_writeprotect) > > #define UFFDIO_CONTINUE _IOWR(UFFDIO, _UFFDIO_CONTINUE, \ > > @@ -222,6 +227,9 @@ struct uffdio_api { > > * asynchronous mode is supported in which the write fault is > > * automatically resolved and write-protection is un-set. > > * It implies UFFD_FEATURE_WP_UNPOPULATED. > > + * > > + * UFFD_FEATURE_MOVE indicates that the kernel supports moving an > > + * existing page contents from userspace. > > */ > > #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) > > #define UFFD_FEATURE_EVENT_FORK (1<<1) > > @@ -239,6 +247,7 @@ struct uffdio_api { > > #define UFFD_FEATURE_WP_UNPOPULATED (1<<13) > > #define UFFD_FEATURE_POISON (1<<14) > > #define UFFD_FEATURE_WP_ASYNC (1<<15) > > +#define UFFD_FEATURE_MOVE (1<<16) > > __u64 features; > > > > __u64 ioctls; > > @@ -347,6 +356,24 @@ struct uffdio_poison { > > __s64 updated; > > }; > > > > +struct uffdio_move { > > + __u64 dst; > > + __u64 src; > > + __u64 len; > > + /* > > + * Especially if used to atomically remove memory from the > > + * address space the wake on the dst range is not needed. > > + */ > > +#define UFFDIO_MOVE_MODE_DONTWAKE ((__u64)1<<0) > > +#define UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES ((__u64)1<<1) > > + __u64 mode; > > + /* > > + * "move" is written by the ioctl and must be at the end: the > > + * copy_from_user will not read the last 8 bytes. > > + */ > > + __s64 move; > > +}; > > + > > /* > > * Flags for the userfaultfd(2) system call itself. > > */ > > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > > index 9656be95a542..6fac5c3d66e6 100644 > > --- a/mm/huge_memory.c > > +++ b/mm/huge_memory.c > > @@ -2086,6 +2086,144 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, > > return ret; > > } > > > > +#ifdef CONFIG_USERFAULTFD > > +/* > > + * The PT lock for src_pmd and the mmap_lock for reading are held by > > + * the caller, but it must return after releasing the > > + * page_table_lock. Just move the page from src_pmd to dst_pmd if possible. > > + * Return zero if succeeded in moving the page, -EAGAIN if it needs to be > > + * repeated by the caller, or other errors in case of failure. > > + */ > > +int remap_pages_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > + pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, > > + struct vm_area_struct *dst_vma, > > + struct vm_area_struct *src_vma, > > + unsigned long dst_addr, unsigned long src_addr) > > +{ > > + pmd_t _dst_pmd, src_pmdval; > > + struct page *src_page; > > + struct folio *src_folio; > > + struct anon_vma *src_anon_vma; > > + spinlock_t *src_ptl, *dst_ptl; > > + pgtable_t src_pgtable, dst_pgtable; > > + struct mmu_notifier_range range; > > + int err = 0; > > + > > + src_pmdval = *src_pmd; > > + src_ptl = pmd_lockptr(src_mm, src_pmd); > > + > > + lockdep_assert_held(src_ptl); > > + mmap_assert_locked(src_mm); > > + mmap_assert_locked(dst_mm); > > + > > + BUG_ON(!pmd_none(dst_pmdval)); > > + BUG_ON(src_addr & ~HPAGE_PMD_MASK); > > + BUG_ON(dst_addr & ~HPAGE_PMD_MASK); > > + > > + if (!pmd_trans_huge(src_pmdval)) { > > + spin_unlock(src_ptl); > > + if (is_pmd_migration_entry(src_pmdval)) { > > + pmd_migration_entry_wait(src_mm, &src_pmdval); > > + return -EAGAIN; > > + } > > + return -ENOENT; > > + } > > + > > + src_page = pmd_page(src_pmdval); > > + if (unlikely(!PageAnonExclusive(src_page))) { > > + spin_unlock(src_ptl); > > + return -EBUSY; > > + } > > + > > + src_folio = page_folio(src_page); > > + folio_get(src_folio); > > + spin_unlock(src_ptl); > > + > > + /* preallocate dst_pgtable if needed */ > > + if (dst_mm != src_mm) { > > + dst_pgtable = pte_alloc_one(dst_mm); > > + if (unlikely(!dst_pgtable)) { > > + err = -ENOMEM; > > + goto put_folio; > > + } > > + } else { > > + dst_pgtable = NULL; > > + } > > + > > IIUC Lokesh's comment applies here, we probably need the > flush_cache_range(), not for x86 but for the other ones.. > > cachetlb.rst: > > Next, we have the cache flushing interfaces. In general, when Linux > is changing an existing virtual-->physical mapping to a new value, > the sequence will be in one of the following forms:: > > 1) flush_cache_mm(mm); > change_all_page_tables_of(mm); > flush_tlb_mm(mm); > > 2) flush_cache_range(vma, start, end); > change_range_of_page_tables(mm, start, end); > flush_tlb_range(vma, start, end); > > 3) flush_cache_page(vma, addr, pfn); > set_pte(pte_pointer, new_pte_val); > flush_tlb_page(vma, addr); Thanks for the reference. I guess that's to support VIVT caches? > > > + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr, > > + src_addr + HPAGE_PMD_SIZE); > > + mmu_notifier_invalidate_range_start(&range); > > + > > + folio_lock(src_folio); > > + > > + /* > > + * split_huge_page walks the anon_vma chain without the page > > + * lock. Serialize against it with the anon_vma lock, the page > > + * lock is not enough. > > + */ > > + src_anon_vma = folio_get_anon_vma(src_folio); > > + if (!src_anon_vma) { > > + err = -EAGAIN; > > + goto unlock_folio; > > + } > > + anon_vma_lock_write(src_anon_vma); > > + > > + dst_ptl = pmd_lockptr(dst_mm, dst_pmd); > > + double_pt_lock(src_ptl, dst_ptl); > > + if (unlikely(!pmd_same(*src_pmd, src_pmdval) || > > + !pmd_same(*dst_pmd, dst_pmdval))) { > > + double_pt_unlock(src_ptl, dst_ptl); > > + err = -EAGAIN; > > + goto put_anon_vma; > > + } > > + if (!PageAnonExclusive(&src_folio->page)) { > > + double_pt_unlock(src_ptl, dst_ptl); > > + err = -EBUSY; > > + goto put_anon_vma; > > + } > > + > > + BUG_ON(!folio_test_head(src_folio)); > > + BUG_ON(!folio_test_anon(src_folio)); > > + > > + folio_move_anon_rmap(src_folio, dst_vma); > > + WRITE_ONCE(src_folio->index, linear_page_index(dst_vma, dst_addr)); > > + > > + src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd); > > + _dst_pmd = mk_huge_pmd(&src_folio->page, dst_vma->vm_page_prot); > > + _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma); > > Last time the conclusion is we leverage can_change_pmd_writable(), no? After your explanation that this works correctly for soft-dirty and UFFD_WP I thought the only thing left to handle was the check for VM_WRITE in both src_vma and dst_vma (which I added into validate_remap_areas()). Maybe I misunderstood and if so, I can replace the above PageAnonExclusive() with can_change_pmd_writable() (note that we err out on VM_SHARED VMAs, so PageAnonExclusive() will be included in that check). > > > + set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd); > > + > > + src_pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd); > > + if (dst_pgtable) { > > + pgtable_trans_huge_deposit(dst_mm, dst_pmd, dst_pgtable); > > + pte_free(src_mm, src_pgtable); > > + dst_pgtable = NULL; > > + > > + mm_inc_nr_ptes(dst_mm); > > + mm_dec_nr_ptes(src_mm); > > + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); > > + add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR); > > + } else { > > + pgtable_trans_huge_deposit(dst_mm, dst_pmd, src_pgtable); > > + } > > + double_pt_unlock(src_ptl, dst_ptl); > > + > > +put_anon_vma: > > + anon_vma_unlock_write(src_anon_vma); > > + put_anon_vma(src_anon_vma); > > +unlock_folio: > > + /* unblock rmap walks */ > > + folio_unlock(src_folio); > > + mmu_notifier_invalidate_range_end(&range); > > + if (dst_pgtable) > > + pte_free(dst_mm, dst_pgtable); > > +put_folio: > > + folio_put(src_folio); > > + > > + return err; > > +} > > +#endif /* CONFIG_USERFAULTFD */ > > + > > /* > > * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise. > > * > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > > index 2b5c0321d96b..0c1ee7172852 100644 > > --- a/mm/khugepaged.c > > +++ b/mm/khugepaged.c > > @@ -1136,6 +1136,9 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, > > * Prevent all access to pagetables with the exception of > > * gup_fast later handled by the ptep_clear_flush and the VM > > * handled by the anon_vma lock + PG_lock. > > + * > > + * UFFDIO_MOVE is prevented to race as well thanks to the > > + * mmap_lock. > > */ > > mmap_write_lock(mm); > > result = hugepage_vma_revalidate(mm, address, true, &vma, cc); > > diff --git a/mm/rmap.c b/mm/rmap.c > > index f9ddc50269d2..a5919cac9a08 100644 > > --- a/mm/rmap.c > > +++ b/mm/rmap.c > > @@ -490,6 +490,12 @@ void __init anon_vma_init(void) > > * page_remove_rmap() that the anon_vma pointer from page->mapping is valid > > * if there is a mapcount, we can dereference the anon_vma after observing > > * those. > > + * > > + * NOTE: the caller should normally hold folio lock when calling this. If > > + * not, the caller needs to double check the anon_vma didn't change after > > + * taking the anon_vma lock for either read or write (UFFDIO_MOVE can modify it > > + * concurrently without folio lock protection). See folio_lock_anon_vma_read() > > + * which has already covered that, and comment above remap_pages(). > > */ > > struct anon_vma *folio_get_anon_vma(struct folio *folio) > > { > > diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c > > index 96d9eae5c7cc..45ce1a8b8ab9 100644 > > --- a/mm/userfaultfd.c > > +++ b/mm/userfaultfd.c > > @@ -842,3 +842,605 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, > > mmap_read_unlock(dst_mm); > > return err; > > } > > + > > + > > +void double_pt_lock(spinlock_t *ptl1, > > + spinlock_t *ptl2) > > + __acquires(ptl1) > > + __acquires(ptl2) > > +{ > > + spinlock_t *ptl_tmp; > > + > > + if (ptl1 > ptl2) { > > + /* exchange ptl1 and ptl2 */ > > + ptl_tmp = ptl1; > > + ptl1 = ptl2; > > + ptl2 = ptl_tmp; > > + } > > + /* lock in virtual address order to avoid lock inversion */ > > + spin_lock(ptl1); > > + if (ptl1 != ptl2) > > + spin_lock_nested(ptl2, SINGLE_DEPTH_NESTING); > > + else > > + __acquire(ptl2); > > +} > > + > > +void double_pt_unlock(spinlock_t *ptl1, > > + spinlock_t *ptl2) > > + __releases(ptl1) > > + __releases(ptl2) > > +{ > > + spin_unlock(ptl1); > > + if (ptl1 != ptl2) > > + spin_unlock(ptl2); > > + else > > + __release(ptl2); > > +} > > + > > + > > +static int remap_present_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > + struct vm_area_struct *dst_vma, > > + struct vm_area_struct *src_vma, > > + unsigned long dst_addr, unsigned long src_addr, > > + pte_t *dst_pte, pte_t *src_pte, > > + pte_t orig_dst_pte, pte_t orig_src_pte, > > + spinlock_t *dst_ptl, spinlock_t *src_ptl, > > + struct folio *src_folio) > > +{ > > + double_pt_lock(dst_ptl, src_ptl); > > + > > + if (!pte_same(*src_pte, orig_src_pte) || > > + !pte_same(*dst_pte, orig_dst_pte)) { > > + double_pt_unlock(dst_ptl, src_ptl); > > + return -EAGAIN; > > + } > > + if (folio_test_large(src_folio) || > > + !PageAnonExclusive(&src_folio->page)) { > > + double_pt_unlock(dst_ptl, src_ptl); > > + return -EBUSY; > > + } > > + > > + BUG_ON(!folio_test_anon(src_folio)); > > + > > + folio_move_anon_rmap(src_folio, dst_vma); > > + WRITE_ONCE(src_folio->index, linear_page_index(dst_vma, dst_addr)); > > + > > + orig_src_pte = ptep_clear_flush(src_vma, src_addr, src_pte); > > + orig_dst_pte = mk_pte(&src_folio->page, dst_vma->vm_page_prot); > > + orig_dst_pte = maybe_mkwrite(pte_mkdirty(orig_dst_pte), dst_vma); > > can_change_pte_writable()? Same as my previous comment. If that's still needed I'll replace the above PageAnonExclusive() check with can_change_pte_writable(). > > > + > > + set_pte_at(dst_mm, dst_addr, dst_pte, orig_dst_pte); > > + > > + if (dst_mm != src_mm) { > > + inc_mm_counter(dst_mm, MM_ANONPAGES); > > + dec_mm_counter(src_mm, MM_ANONPAGES); > > + } > > + > > + double_pt_unlock(dst_ptl, src_ptl); > > + > > + return 0; > > +} > > + > > +static int remap_swap_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > + unsigned long dst_addr, unsigned long src_addr, > > + pte_t *dst_pte, pte_t *src_pte, > > + pte_t orig_dst_pte, pte_t orig_src_pte, > > + spinlock_t *dst_ptl, spinlock_t *src_ptl) > > +{ > > + if (!pte_swp_exclusive(orig_src_pte)) > > + return -EBUSY; > > + > > + double_pt_lock(dst_ptl, src_ptl); > > + > > + if (!pte_same(*src_pte, orig_src_pte) || > > + !pte_same(*dst_pte, orig_dst_pte)) { > > + double_pt_unlock(dst_ptl, src_ptl); > > + return -EAGAIN; > > + } > > + > > + orig_src_pte = ptep_get_and_clear(src_mm, src_addr, src_pte); > > + set_pte_at(dst_mm, dst_addr, dst_pte, orig_src_pte); > > + > > + if (dst_mm != src_mm) { > > + inc_mm_counter(dst_mm, MM_SWAPENTS); > > + dec_mm_counter(src_mm, MM_SWAPENTS); > > + } > > + > > + double_pt_unlock(dst_ptl, src_ptl); > > + > > + return 0; > > +} > > + > > +/* > > + * The mmap_lock for reading is held by the caller. Just move the page > > + * from src_pmd to dst_pmd if possible, and return true if succeeded > > + * in moving the page. > > + */ > > +static int remap_pages_pte(struct mm_struct *dst_mm, > > + struct mm_struct *src_mm, > > + pmd_t *dst_pmd, > > + pmd_t *src_pmd, > > + struct vm_area_struct *dst_vma, > > + struct vm_area_struct *src_vma, > > + unsigned long dst_addr, > > + unsigned long src_addr, > > + __u64 mode) > > +{ > > + swp_entry_t entry; > > + pte_t orig_src_pte, orig_dst_pte; > > + pte_t src_folio_pte; > > + spinlock_t *src_ptl, *dst_ptl; > > + pte_t *src_pte = NULL; > > + pte_t *dst_pte = NULL; > > + > > + struct folio *src_folio = NULL; > > + struct anon_vma *src_anon_vma = NULL; > > + struct mmu_notifier_range range; > > + int err = 0; > > + > > Same flush_cache_range(). Ack. > > > + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, > > + src_addr, src_addr + PAGE_SIZE); > > + mmu_notifier_invalidate_range_start(&range); > > +retry: > > + dst_pte = pte_offset_map_nolock(dst_mm, dst_pmd, dst_addr, &dst_ptl); > > + > > + /* Retry if a huge pmd materialized from under us */ > > + if (unlikely(!dst_pte)) { > > + err = -EAGAIN; > > + goto out; > > + } > > + > > + src_pte = pte_offset_map_nolock(src_mm, src_pmd, src_addr, &src_ptl); > > + > > + /* > > + * We held the mmap_lock for reading so MADV_DONTNEED > > + * can zap transparent huge pages under us, or the > > + * transparent huge page fault can establish new > > + * transparent huge pages under us. > > + */ > > + if (unlikely(!src_pte)) { > > + err = -EAGAIN; > > + goto out; > > + } > > + > > + BUG_ON(pmd_none(*dst_pmd)); > > + BUG_ON(pmd_none(*src_pmd)); > > + BUG_ON(pmd_trans_huge(*dst_pmd)); > > + BUG_ON(pmd_trans_huge(*src_pmd)); > > + > > + spin_lock(dst_ptl); > > + orig_dst_pte = *dst_pte; > > + spin_unlock(dst_ptl); > > + if (!pte_none(orig_dst_pte)) { > > + err = -EEXIST; > > + goto out; > > + } > > + > > + spin_lock(src_ptl); > > + orig_src_pte = *src_pte; > > + spin_unlock(src_ptl); > > + if (pte_none(orig_src_pte)) { > > + if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) > > + err = -ENOENT; > > + else /* nothing to do to remap a hole */ > > + err = 0; > > + goto out; > > + } > > + > > + /* If PTE changed after we locked the folio them start over */ > > + if (src_folio && unlikely(!pte_same(src_folio_pte, orig_src_pte))) { > > + err = -EAGAIN; > > + goto out; > > + } > > + > > + if (pte_present(orig_src_pte)) { > > + /* > > + * Pin and lock both source folio and anon_vma. Since we are in > > + * RCU read section, we can't block, so on contention have to > > + * unmap the ptes, obtain the lock and retry. > > + */ > > + if (!src_folio) { > > + struct folio *folio; > > + > > + /* > > + * Pin the page while holding the lock to be sure the > > + * page isn't freed under us > > + */ > > + spin_lock(src_ptl); > > + if (!pte_same(orig_src_pte, *src_pte)) { > > + spin_unlock(src_ptl); > > + err = -EAGAIN; > > + goto out; > > + } > > + > > + folio = vm_normal_folio(src_vma, src_addr, orig_src_pte); > > + if (!folio || folio_test_large(folio) || > > + !PageAnonExclusive(&folio->page)) { > > + spin_unlock(src_ptl); > > + err = -EBUSY; > > + goto out; > > + } > > + > > + folio_get(folio); > > + src_folio = folio; > > + src_folio_pte = orig_src_pte; > > + spin_unlock(src_ptl); > > + > > + if (!folio_trylock(src_folio)) { > > + pte_unmap(&orig_src_pte); > > + pte_unmap(&orig_dst_pte); > > + src_pte = dst_pte = NULL; > > + /* now we can block and wait */ > > + folio_lock(src_folio); > > + goto retry; > > + } > > + } > > + > > + if (!src_anon_vma) { > > + /* > > + * folio_referenced walks the anon_vma chain > > + * without the folio lock. Serialize against it with > > + * the anon_vma lock, the folio lock is not enough. > > + */ > > + src_anon_vma = folio_get_anon_vma(src_folio); > > + if (!src_anon_vma) { > > + /* page was unmapped from under us */ > > + err = -EAGAIN; > > + goto out; > > + } > > + if (!anon_vma_trylock_write(src_anon_vma)) { > > + pte_unmap(&orig_src_pte); > > + pte_unmap(&orig_dst_pte); > > + src_pte = dst_pte = NULL; > > + /* now we can block and wait */ > > + anon_vma_lock_write(src_anon_vma); > > + goto retry; > > + } > > + } > > + > > + err = remap_present_pte(dst_mm, src_mm, dst_vma, src_vma, > > + dst_addr, src_addr, dst_pte, src_pte, > > + orig_dst_pte, orig_src_pte, > > + dst_ptl, src_ptl, src_folio); > > + } else { > > + entry = pte_to_swp_entry(orig_src_pte); > > + if (non_swap_entry(entry)) { > > + if (is_migration_entry(entry)) { > > + pte_unmap(&orig_src_pte); > > + pte_unmap(&orig_dst_pte); > > + src_pte = dst_pte = NULL; > > + migration_entry_wait(src_mm, src_pmd, > > + src_addr); > > + err = -EAGAIN; > > + } else > > + err = -EFAULT; > > + goto out; > > + } > > + > > + err = remap_swap_pte(dst_mm, src_mm, dst_addr, src_addr, > > + dst_pte, src_pte, > > + orig_dst_pte, orig_src_pte, > > + dst_ptl, src_ptl); > > + } > > + > > +out: > > + if (src_anon_vma) { > > + anon_vma_unlock_write(src_anon_vma); > > + put_anon_vma(src_anon_vma); > > + } > > + if (src_folio) { > > + folio_unlock(src_folio); > > + folio_put(src_folio); > > + } > > + if (dst_pte) > > + pte_unmap(dst_pte); > > + if (src_pte) > > + pte_unmap(src_pte); > > + mmu_notifier_invalidate_range_end(&range); > > + > > + return err; > > +} > > + > > +static int validate_remap_areas(struct vm_area_struct *src_vma, > > s/remap/move/ Ack. > > > + struct vm_area_struct *dst_vma) > > +{ > > + /* Only allow remapping if both have the same access and protection */ > > s/remap/move/ Ack. > > PS: I'll stop comment on renamings.. but please have a look over the whole > patch.. Will do. > > > + if ((src_vma->vm_flags & VM_ACCESS_FLAGS) != (dst_vma->vm_flags & VM_ACCESS_FLAGS) || > > + pgprot_val(src_vma->vm_page_prot) != pgprot_val(dst_vma->vm_page_prot)) > > + return -EINVAL; > > + > > + /* Only allow remapping if both are mlocked or both aren't */ > > + if ((src_vma->vm_flags & VM_LOCKED) != (dst_vma->vm_flags & VM_LOCKED)) > > + return -EINVAL; > > + > > + if (!(src_vma->vm_flags & VM_WRITE) || !(dst_vma->vm_flags & VM_WRITE)) > > + return -EINVAL; > > + > > + /* > > + * Be strict and only allow remap_pages if either the src or > > + * dst range is registered in the userfaultfd to prevent > > + * userland errors going unnoticed. As far as the VM > > + * consistency is concerned, it would be perfectly safe to > > + * remove this check, but there's no useful usage for > > + * remap_pages ouside of userfaultfd registered ranges. This > > + * is after all why it is an ioctl belonging to the > > + * userfaultfd and not a syscall. > > + * > > + * Allow both vmas to be registered in the userfaultfd, just > > + * in case somebody finds a way to make such a case useful. > > + * Normally only one of the two vmas would be registered in > > + * the userfaultfd. > > + */ > > + if (!dst_vma->vm_userfaultfd_ctx.ctx && > > + !src_vma->vm_userfaultfd_ctx.ctx) > > + return -EINVAL; > > When rethinking this, I'm wondering whether we should make it strict. > > The problem is, src/dst is not equal in this case. We got dst_mm from the > uffd context passed in, so we know at least dst_mm has something registered > onto this uffd we're working on, no matter it's the dst_vma or not. It'll > be weird if dst_vma is not registered to the uffd context we're using right > now.. > > Then, can we have a case where src vma is registered with uffd, then dest > vma is not? It sounds really weird, since dest mm needs to be registered > to uffd anyway. It sounds to me the interface is a bit loose and unclear. > > I'm thinking a more reasonable way to safe check this: we model UFFDIO_MOVE > the same way as UFFDIO_COPY/CONTINUE/... where we always assumed the uffd > context is the target (uffd registration required), while current mm is > always the source (no uffd registration required). > > It also means "dropping current page" does not require privilege (here > UFFDIO_MOVE is the same as DONTNEED, per discussed), while "installing a > page into the pgtable" does require some privilege (where privilege granted > by the uffd itself). > > Then it means something like: > > if (!dst_vma->vm_userfaultfd_ctx.ctx || > dst_vma->vm_userfaultfd_ctx.ctx != ctx) > return -EINVAL; > > Here ctx should be passed over from upper. > > What do you think? I didn't think that hard about this check but with your analysis it does sound reasonable. I'll adopt it. Thanks! > > PS: it seems current uffd ioctls are not double checking the vma's ctx is > the same as the ctx that the ioctl is operating on.. but it seems we should > do that for all. I'll think more about it.. > > > + > > + /* > > + * FIXME: only allow remapping across anonymous vmas, > > + * tmpfs should be added. > > + */ > > + if (!vma_is_anonymous(src_vma) || !vma_is_anonymous(dst_vma)) > > + return -EINVAL; > > + > > + /* > > + * Ensure the dst_vma has a anon_vma or this page > > + * would get a NULL anon_vma when moved in the > > + * dst_vma. > > + */ > > + if (unlikely(anon_vma_prepare(dst_vma))) > > + return -ENOMEM; > > + > > + return 0; > > +} > > + > > +/** > > + * remap_pages - remap arbitrary anonymous pages of an existing vma > > + * @dst_start: start of the destination virtual memory range > > + * @src_start: start of the source virtual memory range > > + * @len: length of the virtual memory range > > + * > > + * remap_pages() remaps arbitrary anonymous pages atomically in zero > > + * copy. It only works on non shared anonymous pages because those can > > + * be relocated without generating non linear anon_vmas in the rmap > > + * code. > > + * > > + * It provides a zero copy mechanism to handle userspace page faults. > > + * The source vma pages should have mapcount == 1, which can be > > + * enforced by using madvise(MADV_DONTFORK) on src vma. > > + * > > + * The thread receiving the page during the userland page fault > > + * will receive the faulting page in the source vma through the network, > > + * storage or any other I/O device (MADV_DONTFORK in the source vma > > + * avoids remap_pages() to fail with -EBUSY if the process forks before > > + * remap_pages() is called), then it will call remap_pages() to map the > > + * page in the faulting address in the destination vma. > > + * > > + * This userfaultfd command works purely via pagetables, so it's the > > + * most efficient way to move physical non shared anonymous pages > > + * across different virtual addresses. Unlike mremap()/mmap()/munmap() > > + * it does not create any new vmas. The mapping in the destination > > + * address is atomic. > > + * > > + * It only works if the vma protection bits are identical from the > > + * source and destination vma. > > + * > > + * It can remap non shared anonymous pages within the same vma too. > > + * > > + * If the source virtual memory range has any unmapped holes, or if > > + * the destination virtual memory range is not a whole unmapped hole, > > + * remap_pages() will fail respectively with -ENOENT or -EEXIST. This > > + * provides a very strict behavior to avoid any chance of memory > > + * corruption going unnoticed if there are userland race conditions. > > + * Only one thread should resolve the userland page fault at any given > > + * time for any given faulting address. This means that if two threads > > + * try to both call remap_pages() on the same destination address at the > > + * same time, the second thread will get an explicit error from this > > + * command. > > + * > > + * The command retval will return "len" is successful. The command > > + * however can be interrupted by fatal signals or errors. If > > + * interrupted it will return the number of bytes successfully > > + * remapped before the interruption if any, or the negative error if > > + * none. It will never return zero. Either it will return an error or > > + * an amount of bytes successfully moved. If the retval reports a > > + * "short" remap, the remap_pages() command should be repeated by > > + * userland with src+retval, dst+reval, len-retval if it wants to know > > + * about the error that interrupted it. > > + * > > + * The UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES flag can be specified to > > + * prevent -ENOENT errors to materialize if there are holes in the > > + * source virtual range that is being remapped. The holes will be > > + * accounted as successfully remapped in the retval of the > > + * command. This is mostly useful to remap hugepage naturally aligned > > + * virtual regions without knowing if there are transparent hugepage > > + * in the regions or not, but preventing the risk of having to split > > + * the hugepmd during the remap. > > + * > > + * If there's any rmap walk that is taking the anon_vma locks without > > + * first obtaining the folio lock (the only current instance is > > + * folio_referenced), they will have to verify if the folio->mapping > > + * has changed after taking the anon_vma lock. If it changed they > > + * should release the lock and retry obtaining a new anon_vma, because > > + * it means the anon_vma was changed by remap_pages() before the lock > > + * could be obtained. This is the only additional complexity added to > > + * the rmap code to provide this anonymous page remapping functionality. > > + */ > > +ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > + unsigned long dst_start, unsigned long src_start, > > + unsigned long len, __u64 mode) > > +{ > > + struct vm_area_struct *src_vma, *dst_vma; > > + unsigned long src_addr, dst_addr; > > + pmd_t *src_pmd, *dst_pmd; > > + long err = -EINVAL; > > + ssize_t moved = 0; > > + > > + /* > > + * Sanitize the command parameters: > > + */ > > + BUG_ON(src_start & ~PAGE_MASK); > > + BUG_ON(dst_start & ~PAGE_MASK); > > + BUG_ON(len & ~PAGE_MASK); > > + > > + /* Does the address range wrap, or is the span zero-sized? */ > > + BUG_ON(src_start + len <= src_start); > > + BUG_ON(dst_start + len <= dst_start); > > + > > + /* > > + * Because these are read sempahores there's no risk of lock > > + * inversion. > > + */ > > + mmap_read_lock(dst_mm); > > + if (dst_mm != src_mm) > > + mmap_read_lock(src_mm); > > + > > + /* > > + * Make sure the vma is not shared, that the src and dst remap > > + * ranges are both valid and fully within a single existing > > + * vma. > > + */ > > + src_vma = find_vma(src_mm, src_start); > > + if (!src_vma || (src_vma->vm_flags & VM_SHARED)) > > + goto out; > > + if (src_start < src_vma->vm_start || > > + src_start + len > src_vma->vm_end) > > + goto out; > > + > > + dst_vma = find_vma(dst_mm, dst_start); > > + if (!dst_vma || (dst_vma->vm_flags & VM_SHARED)) > > + goto out; > > + if (dst_start < dst_vma->vm_start || > > + dst_start + len > dst_vma->vm_end) > > + goto out; > > + > > + err = validate_remap_areas(src_vma, dst_vma); > > + if (err) > > + goto out; > > + > > + for (src_addr = src_start, dst_addr = dst_start; > > + src_addr < src_start + len;) { > > + spinlock_t *ptl; > > + pmd_t dst_pmdval; > > + unsigned long step_size; > > + > > + BUG_ON(dst_addr >= dst_start + len); > > + /* > > + * Below works because anonymous area would not have a > > + * transparent huge PUD. If file-backed support is added, > > + * that case would need to be handled here. > > + */ > > + src_pmd = mm_find_pmd(src_mm, src_addr); > > + if (unlikely(!src_pmd)) { > > + if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) { > > + err = -ENOENT; > > + break; > > + } > > + src_pmd = mm_alloc_pmd(src_mm, src_addr); > > + if (unlikely(!src_pmd)) { > > + err = -ENOMEM; > > + break; > > + } > > + } > > + dst_pmd = mm_alloc_pmd(dst_mm, dst_addr); > > + if (unlikely(!dst_pmd)) { > > + err = -ENOMEM; > > + break; > > + } > > + > > + dst_pmdval = pmdp_get_lockless(dst_pmd); > > + /* > > + * If the dst_pmd is mapped as THP don't override it and just > > + * be strict. If dst_pmd changes into TPH after this check, the > > + * remap_pages_huge_pmd() will detect the change and retry > > + * while remap_pages_pte() will detect the change and fail. > > + */ > > + if (unlikely(pmd_trans_huge(dst_pmdval))) { > > + err = -EEXIST; > > + break; > > + } > > + > > + ptl = pmd_trans_huge_lock(src_pmd, src_vma); > > + if (ptl) { > > + if (pmd_devmap(*src_pmd)) { > > + spin_unlock(ptl); > > + err = -ENOENT; > > + break; > > + } > > + > > + /* > > + * Check if we can move the pmd without > > + * splitting it. First check the address > > + * alignment to be the same in src/dst. These > > + * checks don't actually need the PT lock but > > + * it's good to do it here to optimize this > > + * block away at build time if > > + * CONFIG_TRANSPARENT_HUGEPAGE is not set. > > + */ > > + if ((src_addr & ~HPAGE_PMD_MASK) || (dst_addr & ~HPAGE_PMD_MASK) || > > + src_start + len - src_addr < HPAGE_PMD_SIZE || !pmd_none(dst_pmdval)) { > > + spin_unlock(ptl); > > + split_huge_pmd(src_vma, src_pmd, src_addr); > > + continue; > > + } > > + > > + err = remap_pages_huge_pmd(dst_mm, src_mm, > > + dst_pmd, src_pmd, > > + dst_pmdval, > > + dst_vma, src_vma, > > + dst_addr, src_addr); > > + step_size = HPAGE_PMD_SIZE; > > + } else { > > + if (pmd_none(*src_pmd)) { > > + if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) { > > + err = -ENOENT; > > + break; > > + } > > + if (unlikely(__pte_alloc(src_mm, src_pmd))) { > > + err = -ENOMEM; > > + break; > > + } > > + } > > + > > + if (unlikely(pte_alloc(dst_mm, dst_pmd))) { > > + err = -ENOMEM; > > + break; > > + } > > + > > + err = remap_pages_pte(dst_mm, src_mm, > > + dst_pmd, src_pmd, > > + dst_vma, src_vma, > > + dst_addr, src_addr, > > + mode); > > + step_size = PAGE_SIZE; > > + } > > + > > + cond_resched(); > > + > > + if (fatal_signal_pending(current)) { > > + /* Do not override an error */ > > + if (!err || err == -EAGAIN) > > + err = -EINTR; > > + break; > > + } > > + > > + if (err) { > > + if (err == -EAGAIN) > > + continue; > > + break; > > + } > > + > > + /* Proceed to the next page */ > > + dst_addr += step_size; > > + src_addr += step_size; > > + moved += step_size; > > + } > > + > > +out: > > + mmap_read_unlock(dst_mm); > > + if (dst_mm != src_mm) > > + mmap_read_unlock(src_mm); > > + BUG_ON(moved < 0); > > + BUG_ON(err > 0); > > + BUG_ON(!moved && !err); > > + return moved ? moved : err; > > +} > > -- > > 2.42.0.609.gbb76f46606-goog > > > > -- > Peter Xu >
On Fri, Oct 13, 2023 at 9:08 AM Peter Xu <peterx@redhat.com> wrote: > > On Fri, Oct 13, 2023 at 11:56:31AM +0200, David Hildenbrand wrote: > > Hi Peter, > > Hi, David, > > > > > > I used to have the same thought with David on whether we can simplify the > > > design to e.g. limit it to single mm. Then I found that the trickiest is > > > actually patch 1 together with the anon_vma manipulations, and the problem > > > is that's not avoidable even if we restrict the api to apply on single mm. > > > > > > What else we can benefit from single mm? One less mmap read lock, but > > > probably that's all we can get; IIUC we need to keep most of the rest of > > > the code, e.g. pgtable walks, double pgtable lockings, etc. > > > > No existing mechanisms move anon pages between unrelated processes, that > > naturally makes me nervous if we're doing it "just because we can". > > IMHO that's also the potential, when guarded with userfaultfd descriptor > being shared between two processes. > > See below with more comment on the raised concerns. > > > > > > > > > Actually, even though I have no solid clue, but I had a feeling that there > > > can be some interesting way to leverage this across-mm movement, while > > > keeping things all safe (by e.g. elaborately requiring other proc to create > > > uffd and deliver to this proc). > > > > Okay, but no real use cases yet. > > I can provide a "not solid" example. I didn't mention it because it's > really something that just popped into my mind when thinking cross-mm, so I > never discussed with anyone yet nor shared it anywhere. > > Consider VM live upgrade in a generic form (e.g., no VFIO), we can do that > very efficiently with shmem or hugetlbfs, but not yet anonymous. We can do > extremely efficient postcopy live upgrade now with anonymous if with REMAP. > > Basically I see it a potential way of moving memory efficiently especially > with thp. > > > > > > > > > Considering Andrea's original version already contains those bits and all > > > above, I'd vote that we go ahead with supporting two MMs. > > > > You can do nasty things with that, as it stands, on the upstream codebase. > > > > If you pin the page in src_mm and move it to dst_mm, you successfully broke > > an invariant that "exclusive" means "no other references from other > > processes". That page is marked exclusive but it is, in fact, not exclusive. > > It is still exclusive to the dst mm? I see your point, but I think you're > taking exclusiveness altogether with pinning, and IMHO that may not be > always necessary? > > > > > Once you achieved that, you can easily have src_mm not have MMF_HAS_PINNED, > > (I suppose you meant dst_mm here) > > > so you can just COW-share that page. Now you successfully broke the > > invariant that COW-shared pages must not be pinned. And you can even trigger > > VM_BUG_ONs, like in sanity_check_pinned_pages(). > > Yeah, that's really unfortunate. But frankly, I don't think it's the fault > of this new feature, but the rest. > > Let's imagine if the MMF_HAS_PINNED wasn't proposed as a per-mm flag, but > per-vma, which I don't see why we can't because it's simply a hint so far. > Then if we apply the same rule here, UFFDIO_REMAP won't even work for > single-mm as long as cross-vma. Then UFFDIO_REMAP as a whole feature will > be NACKed simply because of this.. > > And I don't think anyone can guarantee a per-vma MMF_HAS_PINNED can never > happen, or any further change to pinning solution that may affect this. So > far it just looks unsafe to remap a pin page to me. > > I don't have a good suggestion here if this is a risk.. I'd think it risky > then to do REMAP over pinned pages no matter cross-mm or single-mm. It > means probably we just rule them out: folio_maybe_dma_pinned() may not even > be enough to be safe with fast-gup. We may need page_needs_cow_for_dma() > with proper write_protect_seq no matter cross-mm or single-mm? > > > > > Can it all be fixed? Sure, with more complexity. For something without clear > > motivation, I'll have to pass. > > I think what you raised is a valid concern, but IMHO it's better fixed no > matter cross-mm or single-mm. What do you think? > > In general, pinning lose its whole point here to me for an userspace either > if it DONTNEEDs it or REMAP it. What would be great to do here is we unpin > it upon DONTNEED/REMAP/whatever drops the page, because it loses its > coherency anyway, IMHO. > > > > > Once there is real demand, we can revisit it and explore what else we would > > have to take care of (I don't know how memcg behaves when moving between > > completely unrelated processes, maybe that works as expected, I don't know > > and I have no time to spare on reviewing features with no real use cases) > > and announce it as a new feature. > > Good point. memcg is probably needed.. > > So you reminded me to do a more thorough review against zap/fault paths, I > think what's missing are (besides page pinning): > > - mem_cgroup_charge()/mem_cgroup_uncharge(): Good point. Will add in the next version. > > (side note: I think folio_throttle_swaprate() is only for when > allocating new pages, so not needed here) > > - check_stable_address_space() (under pgtable lock) Ack. > > - tlb flush > > Hmm???????????????? I can't see anywhere we did tlb flush, batched or > not, either single-mm or cross-mm should need it. Is this missing? As Lokesh pointed out we do that but we don't batch them. I'll try to add batching in the next version. > > > > > > > Note: that (with only reading the documentation) it also kept me wondering > > how the MMs are even implied from > > > > struct uffdio_move { > > __u64 dst; /* Destination of move */ > > __u64 src; /* Source of move */ > > __u64 len; /* Number of bytes to move */ > > __u64 mode; /* Flags controlling behavior of move */ > > __s64 move; /* Number of bytes moved, or negated error */ > > }; > > > > That probably has to be documented as well, in which address space dst and > > src reside. > > Agreed, some better documentation will never hurt. Dst should be in the mm > address space that was bound to the userfault descriptor. Src should be in > the current mm address space. Ack. Will add. Thanks! > > Thanks, > > -- > Peter Xu > > -- > To unsubscribe from this group and stop receiving emails from it, send an email to kernel-team+unsubscribe@android.com. >
On Tue, Oct 17, 2023 at 12:40 PM kernel test robot <lkp@intel.com> wrote: > > Hi Suren, > > kernel test robot noticed the following build warnings: > > [auto build test WARNING on akpm-mm/mm-everything] > [also build test WARNING on next-20231017] > [cannot apply to linus/master v6.6-rc6] > [If your patch is applied to the wrong git tree, kindly drop us a note. > And when submitting patch, we suggest to use '--base' as documented in > https://git-scm.com/docs/git-format-patch#_base_tree_information] > > url: https://github.com/intel-lab-lkp/linux/commits/Suren-Baghdasaryan/mm-rmap-support-move-to-different-root-anon_vma-in-folio_move_anon_rmap/20231009-144552 > base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything > patch link: https://lore.kernel.org/r/20231009064230.2952396-3-surenb%40google.com > patch subject: [PATCH v3 2/3] userfaultfd: UFFDIO_MOVE uABI > config: i386-randconfig-141-20231017 (https://download.01.org/0day-ci/archive/20231018/202310180338.zTpcYECK-lkp@intel.com/config) > compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 > reproduce: (https://download.01.org/0day-ci/archive/20231018/202310180338.zTpcYECK-lkp@intel.com/reproduce) > > If you fix the issue in a separate patch/commit (i.e. not just a new version of > the same patch/commit), kindly add following tags > | Reported-by: kernel test robot <lkp@intel.com> > | Closes: https://lore.kernel.org/oe-kbuild-all/202310180338.zTpcYECK-lkp@intel.com/ > > smatch warnings: > mm/userfaultfd.c:1380 remap_pages() warn: unsigned 'src_start + len - src_addr' is never less than zero. Hmm. I think this warning is correct only when CONFIG_TRANSPARENT_HUGEPAGE=n. I guess I'll have to add an "ifdef CONFIG_TRANSPARENT_HUGEPAGE" here after all, which lets us move these checks before locking PTL. > > vim +1380 mm/userfaultfd.c > > 1195 > 1196 /** > 1197 * remap_pages - remap arbitrary anonymous pages of an existing vma > 1198 * @dst_start: start of the destination virtual memory range > 1199 * @src_start: start of the source virtual memory range > 1200 * @len: length of the virtual memory range > 1201 * > 1202 * remap_pages() remaps arbitrary anonymous pages atomically in zero > 1203 * copy. It only works on non shared anonymous pages because those can > 1204 * be relocated without generating non linear anon_vmas in the rmap > 1205 * code. > 1206 * > 1207 * It provides a zero copy mechanism to handle userspace page faults. > 1208 * The source vma pages should have mapcount == 1, which can be > 1209 * enforced by using madvise(MADV_DONTFORK) on src vma. > 1210 * > 1211 * The thread receiving the page during the userland page fault > 1212 * will receive the faulting page in the source vma through the network, > 1213 * storage or any other I/O device (MADV_DONTFORK in the source vma > 1214 * avoids remap_pages() to fail with -EBUSY if the process forks before > 1215 * remap_pages() is called), then it will call remap_pages() to map the > 1216 * page in the faulting address in the destination vma. > 1217 * > 1218 * This userfaultfd command works purely via pagetables, so it's the > 1219 * most efficient way to move physical non shared anonymous pages > 1220 * across different virtual addresses. Unlike mremap()/mmap()/munmap() > 1221 * it does not create any new vmas. The mapping in the destination > 1222 * address is atomic. > 1223 * > 1224 * It only works if the vma protection bits are identical from the > 1225 * source and destination vma. > 1226 * > 1227 * It can remap non shared anonymous pages within the same vma too. > 1228 * > 1229 * If the source virtual memory range has any unmapped holes, or if > 1230 * the destination virtual memory range is not a whole unmapped hole, > 1231 * remap_pages() will fail respectively with -ENOENT or -EEXIST. This > 1232 * provides a very strict behavior to avoid any chance of memory > 1233 * corruption going unnoticed if there are userland race conditions. > 1234 * Only one thread should resolve the userland page fault at any given > 1235 * time for any given faulting address. This means that if two threads > 1236 * try to both call remap_pages() on the same destination address at the > 1237 * same time, the second thread will get an explicit error from this > 1238 * command. > 1239 * > 1240 * The command retval will return "len" is successful. The command > 1241 * however can be interrupted by fatal signals or errors. If > 1242 * interrupted it will return the number of bytes successfully > 1243 * remapped before the interruption if any, or the negative error if > 1244 * none. It will never return zero. Either it will return an error or > 1245 * an amount of bytes successfully moved. If the retval reports a > 1246 * "short" remap, the remap_pages() command should be repeated by > 1247 * userland with src+retval, dst+reval, len-retval if it wants to know > 1248 * about the error that interrupted it. > 1249 * > 1250 * The UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES flag can be specified to > 1251 * prevent -ENOENT errors to materialize if there are holes in the > 1252 * source virtual range that is being remapped. The holes will be > 1253 * accounted as successfully remapped in the retval of the > 1254 * command. This is mostly useful to remap hugepage naturally aligned > 1255 * virtual regions without knowing if there are transparent hugepage > 1256 * in the regions or not, but preventing the risk of having to split > 1257 * the hugepmd during the remap. > 1258 * > 1259 * If there's any rmap walk that is taking the anon_vma locks without > 1260 * first obtaining the folio lock (the only current instance is > 1261 * folio_referenced), they will have to verify if the folio->mapping > 1262 * has changed after taking the anon_vma lock. If it changed they > 1263 * should release the lock and retry obtaining a new anon_vma, because > 1264 * it means the anon_vma was changed by remap_pages() before the lock > 1265 * could be obtained. This is the only additional complexity added to > 1266 * the rmap code to provide this anonymous page remapping functionality. > 1267 */ > 1268 ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm, > 1269 unsigned long dst_start, unsigned long src_start, > 1270 unsigned long len, __u64 mode) > 1271 { > 1272 struct vm_area_struct *src_vma, *dst_vma; > 1273 unsigned long src_addr, dst_addr; > 1274 pmd_t *src_pmd, *dst_pmd; > 1275 long err = -EINVAL; > 1276 ssize_t moved = 0; > 1277 > 1278 /* > 1279 * Sanitize the command parameters: > 1280 */ > 1281 BUG_ON(src_start & ~PAGE_MASK); > 1282 BUG_ON(dst_start & ~PAGE_MASK); > 1283 BUG_ON(len & ~PAGE_MASK); > 1284 > 1285 /* Does the address range wrap, or is the span zero-sized? */ > 1286 BUG_ON(src_start + len <= src_start); > 1287 BUG_ON(dst_start + len <= dst_start); > 1288 > 1289 /* > 1290 * Because these are read sempahores there's no risk of lock > 1291 * inversion. > 1292 */ > 1293 mmap_read_lock(dst_mm); > 1294 if (dst_mm != src_mm) > 1295 mmap_read_lock(src_mm); > 1296 > 1297 /* > 1298 * Make sure the vma is not shared, that the src and dst remap > 1299 * ranges are both valid and fully within a single existing > 1300 * vma. > 1301 */ > 1302 src_vma = find_vma(src_mm, src_start); > 1303 if (!src_vma || (src_vma->vm_flags & VM_SHARED)) > 1304 goto out; > 1305 if (src_start < src_vma->vm_start || > 1306 src_start + len > src_vma->vm_end) > 1307 goto out; > 1308 > 1309 dst_vma = find_vma(dst_mm, dst_start); > 1310 if (!dst_vma || (dst_vma->vm_flags & VM_SHARED)) > 1311 goto out; > 1312 if (dst_start < dst_vma->vm_start || > 1313 dst_start + len > dst_vma->vm_end) > 1314 goto out; > 1315 > 1316 err = validate_remap_areas(src_vma, dst_vma); > 1317 if (err) > 1318 goto out; > 1319 > 1320 for (src_addr = src_start, dst_addr = dst_start; > 1321 src_addr < src_start + len;) { > 1322 spinlock_t *ptl; > 1323 pmd_t dst_pmdval; > 1324 unsigned long step_size; > 1325 > 1326 BUG_ON(dst_addr >= dst_start + len); > 1327 /* > 1328 * Below works because anonymous area would not have a > 1329 * transparent huge PUD. If file-backed support is added, > 1330 * that case would need to be handled here. > 1331 */ > 1332 src_pmd = mm_find_pmd(src_mm, src_addr); > 1333 if (unlikely(!src_pmd)) { > 1334 if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) { > 1335 err = -ENOENT; > 1336 break; > 1337 } > 1338 src_pmd = mm_alloc_pmd(src_mm, src_addr); > 1339 if (unlikely(!src_pmd)) { > 1340 err = -ENOMEM; > 1341 break; > 1342 } > 1343 } > 1344 dst_pmd = mm_alloc_pmd(dst_mm, dst_addr); > 1345 if (unlikely(!dst_pmd)) { > 1346 err = -ENOMEM; > 1347 break; > 1348 } > 1349 > 1350 dst_pmdval = pmdp_get_lockless(dst_pmd); > 1351 /* > 1352 * If the dst_pmd is mapped as THP don't override it and just > 1353 * be strict. If dst_pmd changes into TPH after this check, the > 1354 * remap_pages_huge_pmd() will detect the change and retry > 1355 * while remap_pages_pte() will detect the change and fail. > 1356 */ > 1357 if (unlikely(pmd_trans_huge(dst_pmdval))) { > 1358 err = -EEXIST; > 1359 break; > 1360 } > 1361 > 1362 ptl = pmd_trans_huge_lock(src_pmd, src_vma); > 1363 if (ptl) { > 1364 if (pmd_devmap(*src_pmd)) { > 1365 spin_unlock(ptl); > 1366 err = -ENOENT; > 1367 break; > 1368 } > 1369 > 1370 /* > 1371 * Check if we can move the pmd without > 1372 * splitting it. First check the address > 1373 * alignment to be the same in src/dst. These > 1374 * checks don't actually need the PT lock but > 1375 * it's good to do it here to optimize this > 1376 * block away at build time if > 1377 * CONFIG_TRANSPARENT_HUGEPAGE is not set. > 1378 */ > 1379 if ((src_addr & ~HPAGE_PMD_MASK) || (dst_addr & ~HPAGE_PMD_MASK) || > > 1380 src_start + len - src_addr < HPAGE_PMD_SIZE || !pmd_none(dst_pmdval)) { > > -- > 0-DAY CI Kernel Test Service > https://github.com/intel/lkp-tests/wiki > > -- > To unsubscribe from this group and stop receiving emails from it, send an email to kernel-team+unsubscribe@android.com. >
On 19.10.23 21:53, Peter Xu wrote: > On Thu, Oct 19, 2023 at 05:41:01PM +0200, David Hildenbrand wrote: >> That's not my main point. It can easily become a maintenance burden without >> any real use cases yet that we are willing to support. > > That's why I requested a few times that we can discuss the complexity of > cross-mm support already here, and I'm all ears if I missed something on > the "maintenance burden" part.. > > I started by listing what I think might be different, and we can easily > speedup single-mm with things like "if (ctx->mm != mm)" checks with > e.g. memcg, just like what this patch already did with pgtable depositions. > > We keep saying "maintenance burden" but we refuse to discuss what is that.. Let's recap (1) We have person A up-streaming code written by person B, whereby B is not involved in the discussions nor seems to be active to maintain that code. Worse, the code that is getting up-streamed was originally based on a different kernel version that has significant differences in some key areas -- for example, page pinning, exclusive vs. shared. I claim that nobody here fully understands the code at hand (just look at the previous discussions), and reviewers have to sort out the mess that was created by the very way this stuff is getting upstreamed here. We're already struggling to get the single-mm case working correctly. (2) Cross-mm was not even announced anywhere nor mentioned which use it would have; I had to stumble over this while digging through the code. Further, is it even *tested*? AFAIKS in patch #3 no. Why do we have to make the life of reviewers harder by forcing them to review code that currently *nobody* on this earth needs? (3) You said "What else we can benefit from single mm? One less mmap read lock, but probably that's all we can get;" and I presented two non-obvious issues. I did not even look any further because I really have better things to do than review complicated code without real use cases at hand. As I said "maybe that works as expected, I don't know and I have no time to spare on reviewing features with no real use cases)"; apparently I was right by just guessing that memcg handling is missing. The sub-feature in question (cross-mm) has no solid use cases; at this point I am not even convinced the use case you raised requires *userfaultfd*; for the purpose of moving a whole VMA worth of pages between two processes; I don't see the immediate need to get userfaultfd involved and move individual pages under page lock etc. > > I'll leave that to Suren and Lokesh to decide. For me the worst case is > one more flag which might be confusing, which is not the end of the world.. > Suren, you may need to work more thoroughly to remove cross-mm implications > if so, just like when renaming REMAP to MOVE. I'm asking myself why you are pushing so hard to include complexity "just because we can"; doesn't make any sense to me, honestly. Maybe you have some other real use cases that ultimately require userfaultfd for cross-mm that you cannot share? Will the world end when we have to use a separate flag so we can open this pandora's box when really required? Again, moving anon pages within a process is a known thing; we do that already via mremap; the only difference here really is, that we have to get the rmap right because we don't adjust VMAs. It's a shame we don't try to combine both code paths, maybe it's not easily possible like we did with mprotect vs. uffd-wp. Moving anon pages between process is currently only done via COW, where all things (page pinning, memcg, ...) have been figured out and are simply working as expected. Making uffd special by coding-up their own thing does not sound compelling to me. I am clearly against any unwarranted features+complexity. Again, I will stop arguing further, the whole thing of "include it just because we can" to avoid a flag (that we might never even see) doesn't make any sense to me and likely never will. The whole way this feature is getting upstreamed is just messed up IMHO and I the reasoning used in this thread to stick as-close-as-possible to some code person B wrote some years ago (e.g., naming, sub-features) is far out of my comprehension.
On Fri, Oct 20, 2023 at 3:02 AM David Hildenbrand <david@redhat.com> wrote: > > On 19.10.23 21:53, Peter Xu wrote: > > On Thu, Oct 19, 2023 at 05:41:01PM +0200, David Hildenbrand wrote: > >> That's not my main point. It can easily become a maintenance burden without > >> any real use cases yet that we are willing to support. > > > > That's why I requested a few times that we can discuss the complexity of > > cross-mm support already here, and I'm all ears if I missed something on > > the "maintenance burden" part.. > > > > I started by listing what I think might be different, and we can easily > > speedup single-mm with things like "if (ctx->mm != mm)" checks with > > e.g. memcg, just like what this patch already did with pgtable depositions. > > > > We keep saying "maintenance burden" but we refuse to discuss what is that.. > > Let's recap > > (1) We have person A up-streaming code written by person B, whereby B is > not involved in the discussions nor seems to be active to maintain that > code. > > Worse, the code that is getting up-streamed was originally based on a > different kernel version that has significant differences in some key > areas -- for example, page pinning, exclusive vs. shared. > > I claim that nobody here fully understands the code at hand (just look > at the previous discussions), and reviewers have to sort out the mess > that was created by the very way this stuff is getting upstreamed here. > > We're already struggling to get the single-mm case working correctly. > > > (2) Cross-mm was not even announced anywhere nor mentioned which use it > would have; I had to stumble over this while digging through the code. > Further, is it even *tested*? AFAIKS in patch #3 no. Why do we have to > make the life of reviewers harder by forcing them to review code that > currently *nobody* on this earth needs? > > > (3) You said "What else we can benefit from single mm? One less mmap > read lock, but probably that's all we can get;" and I presented two > non-obvious issues. I did not even look any further because I really > have better things to do than review complicated code without real use > cases at hand. As I said "maybe that works as expected, I > don't know and I have no time to spare on reviewing features with no > real use cases)"; apparently I was right by just guessing that memcg > handling is missing. > > > The sub-feature in question (cross-mm) has no solid use cases; at this > point I am not even convinced the use case you raised requires > *userfaultfd*; for the purpose of moving a whole VMA worth of pages > between two processes; I don't see the immediate need to get userfaultfd > involved and move individual pages under page lock etc. You make a compelling case against cross-mm support. While I can't force Andrea to participate in upstreaming nor do I have his background, keeping it simple, as you requested, is doable. That's what I plan on doing by splitting the patch and I think we all agreed to that. I'll also see if I can easily add a separate patch to test cross-mm support. I do apologize for the extra effort required from reviewers to cover for the gaps in my patches. I'm doing my best to minimize that and I really appreciate your time. > > > > > I'll leave that to Suren and Lokesh to decide. For me the worst case is > > one more flag which might be confusing, which is not the end of the world.. > > Suren, you may need to work more thoroughly to remove cross-mm implications > > if so, just like when renaming REMAP to MOVE. > > I'm asking myself why you are pushing so hard to include complexity > "just because we can"; doesn't make any sense to me, honestly. > > Maybe you have some other real use cases that ultimately require > userfaultfd for cross-mm that you cannot share? > > Will the world end when we have to use a separate flag so we can open > this pandora's box when really required? > > > Again, moving anon pages within a process is a known thing; we do that > already via mremap; the only difference here really is, that we have to > get the rmap right because we don't adjust VMAs. It's a shame we don't > try to combine both code paths, maybe it's not easily possible like we > did with mprotect vs. uffd-wp. That's a good point. With cross-mm support baked in, the overlap was not obvious to me. I'll see how much we can reuse from the mremap path. > > Moving anon pages between process is currently only done via COW, where > all things (page pinning, memcg, ...) have been figured out and are > simply working as expected. Making uffd special by coding-up their own > thing does not sound compelling to me. > > > I am clearly against any unwarranted features+complexity. Again, I will > stop arguing further, the whole thing of "include it just because we > can" to avoid a flag (that we might never even see) doesn't make any > sense to me and likely never will. > > The whole way this feature is getting upstreamed is just messed up IMHO > and I the reasoning used in this thread to stick > as-close-as-possible to some code person B wrote some years ago (e.g., > naming, sub-features) is far out of my comprehension. I don't think staying as-close-as-possible to the original version was the way I was driving this so far. At least that was not my conscious intention. I'm open to further suggestions whenever it makes sense to deviate from it. Thanks, Suren. > > -- > Cheers, > > David / dhildenb >
>> >> The sub-feature in question (cross-mm) has no solid use cases; at this >> point I am not even convinced the use case you raised requires >> *userfaultfd*; for the purpose of moving a whole VMA worth of pages >> between two processes; I don't see the immediate need to get userfaultfd >> involved and move individual pages under page lock etc. > > You make a compelling case against cross-mm support. I tried to :P I'm happy to hear compelling cases for cross-mm support that we need *right now*. And that's what I'm missing so far besides "already included in the patches" and "but we would eventually need a separate flag". As a side note, I already do have another rmap-related feature in the works that will require extra-effort to handle this case (short: assign each MM a unique ID and use that for accounting purposes when (un)mapping pages); I think I figured out how to handle this case here; and it's questionable if my work will make it upstream -- to be posted as PoC in 2-4 weeks I guess. But it easily shows that there are cases where this will require extra work -- without any current benefits due to lack of actual users. > While I can't force Andrea to participate in upstreaming nor do I have > his background, keeping it simple, as you requested, is doable. That's > what I plan on doing by splitting the patch and I think we all agreed > to that. I'll also see if I can easily add a separate patch to test > cross-mm support. > I do apologize for the extra effort required from reviewers to cover > for the gaps in my patches. I'm doing my best to minimize that and I > really appreciate your time. It's absolutely not your fault and there is absolutely no need to apologize (sorry if I sounded like I would be blaming you in any way). I made myself the experience that up-streaming the work of someone else can be troublesome, because it's hard to grasp all the details from a set of patches. Documentation and comments can't handle all the implicit knowledge from the original author. I likely wouldn't be able to even write that code myself. For example: why is cross-mm relevant and was included in the original patches? Maybe there was a very good reason and it is simply not documented. > >> >>> >>> I'll leave that to Suren and Lokesh to decide. For me the worst case is >>> one more flag which might be confusing, which is not the end of the world.. >>> Suren, you may need to work more thoroughly to remove cross-mm implications >>> if so, just like when renaming REMAP to MOVE. >> >> I'm asking myself why you are pushing so hard to include complexity >> "just because we can"; doesn't make any sense to me, honestly. >> >> Maybe you have some other real use cases that ultimately require >> userfaultfd for cross-mm that you cannot share? >> >> Will the world end when we have to use a separate flag so we can open >> this pandora's box when really required? >> >> >> Again, moving anon pages within a process is a known thing; we do that >> already via mremap; the only difference here really is, that we have to >> get the rmap right because we don't adjust VMAs. It's a shame we don't >> try to combine both code paths, maybe it's not easily possible like we >> did with mprotect vs. uffd-wp. > > That's a good point. With cross-mm support baked in, the overlap was > not obvious to me. I'll see how much we can reuse from the mremap > path. My comment was inspired by Lokesh "While going through mremap's move_page_tables code, which is pretty similar to what we do here". There are some subtle differences (could we even move whole page tables? probably not due to holding the mmap locking only in read-mode) and special exclusive-only+rmap adjust handling. Further, TLB flushing is different (but maybe there are ways to just reuse the batching, did not look into the details). But move_page_tables is clearly single-mm code, and a unification might not be that straight forward. > >> >> Moving anon pages between process is currently only done via COW, where >> all things (page pinning, memcg, ...) have been figured out and are >> simply working as expected. Making uffd special by coding-up their own >> thing does not sound compelling to me. >> >> >> I am clearly against any unwarranted features+complexity. Again, I will >> stop arguing further, the whole thing of "include it just because we >> can" to avoid a flag (that we might never even see) doesn't make any >> sense to me and likely never will. >> >> The whole way this feature is getting upstreamed is just messed up IMHO >> and I the reasoning used in this thread to stick >> as-close-as-possible to some code person B wrote some years ago (e.g., >> naming, sub-features) is far out of my comprehension. > > I don't think staying as-close-as-possible to the original version was > the way I was driving this so far. At least that was not my conscious These are rather the vibes I'm getting from Peter. "Why rename it, could confuse people because the original patches are old", "Why exclude it if it has been included in the original patches". Not the kind of reasoning I can relate to when it comes to upstreaming some patches. > intention. I'm open to further suggestions whenever it makes sense to > deviate from it. I'll repeat: any complexity we remove and any code reused in common code/moved out of userfaultfd will be a win.
On Fri, Oct 20, 2023 at 07:16:19PM +0200, David Hildenbrand wrote: > These are rather the vibes I'm getting from Peter. "Why rename it, could > confuse people because the original patches are old", "Why exclude it if it > has been included in the original patches". Not the kind of reasoning I can > relate to when it comes to upstreaming some patches. You can't blame anyone if you misunderstood and biased the question. The first question is definitely valid, even until now. You guys still prefer to rename it, which I'm totally fine with. The 2nd question is wrong from your interpretation. That's not my point, at least not starting from a few replies already. What I was asking for is why such page movement between mm is dangerous. I don't think I get solid answers even until now. Noticing "memcg is missing" is not an argument for "cross-mm is dangerous", it's a review comment. Suren can address that. You'll propose a new feature that may tag an mm is not an argument either, if it's not merged yet. We can also address that depending on what it is, also on which lands earlier. It'll be good to discuss these details even in a single-mm support. Anyone would like to add that can already refer to discussion in this thread. I hope I'm clear.
On Thu, Oct 19, 2023 at 02:24:06PM -0700, Suren Baghdasaryan wrote: > On Thu, Oct 12, 2023 at 3:00 PM Peter Xu <peterx@redhat.com> wrote: > > > > On Sun, Oct 08, 2023 at 11:42:27PM -0700, Suren Baghdasaryan wrote: > > > From: Andrea Arcangeli <aarcange@redhat.com> > > > > > > Implement the uABI of UFFDIO_MOVE ioctl. > > > UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application > > > needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are > > > available (in userspace) for recycling, as is usually the case in heap > > > compaction algorithms, then we can avoid the page allocation and memcpy > > > (done by UFFDIO_COPY). Also, since the pages are recycled in the > > > userspace, we avoid the need to release (via madvise) the pages back to > > > the kernel [2]. > > > We see over 40% reduction (on a Google pixel 6 device) in the compacting > > > thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was > > > measured using a benchmark that emulates a heap compaction implementation > > > using userfaultfd (to allow concurrent accesses by application threads). > > > More details of the usecase are explained in [2]. > > > Furthermore, UFFDIO_MOVE enables moving swapped-out pages without > > > touching them within the same vma. Today, it can only be done by mremap, > > > however it forces splitting the vma. > > > > > > [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ > > > [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ > > > > > > Update for the ioctl_userfaultfd(2) manpage: > > > > > > UFFDIO_MOVE > > > (Since Linux xxx) Move a continuous memory chunk into the > > > userfault registered range and optionally wake up the blocked > > > thread. The source and destination addresses and the number of > > > bytes to move are specified by the src, dst, and len fields of > > > the uffdio_move structure pointed to by argp: > > > > > > struct uffdio_move { > > > __u64 dst; /* Destination of move */ > > > __u64 src; /* Source of move */ > > > __u64 len; /* Number of bytes to move */ > > > __u64 mode; /* Flags controlling behavior of move */ > > > __s64 move; /* Number of bytes moved, or negated error */ > > > }; > > > > > > The following value may be bitwise ORed in mode to change the > > > behavior of the UFFDIO_MOVE operation: > > > > > > UFFDIO_MOVE_MODE_DONTWAKE > > > Do not wake up the thread that waits for page-fault > > > resolution > > > > > > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES > > > Allow holes in the source virtual range that is being moved. > > > When not specified, the holes will result in ENOENT error. > > > When specified, the holes will be accounted as successfully > > > moved memory. This is mostly useful to move hugepage aligned > > > virtual regions without knowing if there are transparent > > > hugepages in the regions or not, but preventing the risk of > > > having to split the hugepage during the operation. > > > > > > The move field is used by the kernel to return the number of > > > bytes that was actually moved, or an error (a negated errno- > > > style value). If the value returned in move doesn't match the > > > value that was specified in len, the operation fails with the > > > error EAGAIN. The move field is output-only; it is not read by > > > the UFFDIO_MOVE operation. > > > > > > The operation may fail for various reasons. Usually, remapping of > > > pages that are not exclusive to the given process fail; once KSM > > > might deduplicate pages or fork() COW-shares pages during fork() > > > with child processes, they are no longer exclusive. Further, the > > > kernel might only perform lightweight checks for detecting whether > > > the pages are exclusive, and return -EBUSY in case that check fails. > > > To make the operation more likely to succeed, KSM should be > > > disabled, fork() should be avoided or MADV_DONTFORK should be > > > configured for the source VMA before fork(). > > > > > > This ioctl(2) operation returns 0 on success. In this case, the > > > entire area was moved. On error, -1 is returned and errno is > > > set to indicate the error. Possible errors include: > > > > > > EAGAIN The number of bytes moved (i.e., the value returned in > > > the move field) does not equal the value that was > > > specified in the len field. > > > > > > EINVAL Either dst or len was not a multiple of the system page > > > size, or the range specified by src and len or dst and len > > > was invalid. > > > > > > EINVAL An invalid bit was specified in the mode field. > > > > > > ENOENT > > > The source virtual memory range has unmapped holes and > > > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. > > > > > > EEXIST > > > The destination virtual memory range is fully or partially > > > mapped. > > > > > > EBUSY > > > The pages in the source virtual memory range are not > > > exclusive to the process. The kernel might only perform > > > lightweight checks for detecting whether the pages are > > > exclusive. To make the operation more likely to succeed, > > > KSM should be disabled, fork() should be avoided or > > > MADV_DONTFORK should be configured for the source virtual > > > memory area before fork(). > > > > > > ENOMEM Allocating memory needed for the operation failed. > > > > > > ESRCH > > > The faulting process has exited at the time of a > > > > Nit pick comment for future man page: there's no faulting process in this > > context. Perhaps "target process"? > > Ack. > > > > > > UFFDIO_MOVE operation. > > > > > > Signed-off-by: Andrea Arcangeli <aarcange@redhat.com> > > > Signed-off-by: Suren Baghdasaryan <surenb@google.com> > > > --- > > > Documentation/admin-guide/mm/userfaultfd.rst | 3 + > > > fs/userfaultfd.c | 63 ++ > > > include/linux/rmap.h | 5 + > > > include/linux/userfaultfd_k.h | 12 + > > > include/uapi/linux/userfaultfd.h | 29 +- > > > mm/huge_memory.c | 138 +++++ > > > mm/khugepaged.c | 3 + > > > mm/rmap.c | 6 + > > > mm/userfaultfd.c | 602 +++++++++++++++++++ > > > 9 files changed, 860 insertions(+), 1 deletion(-) > > > > > > diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst > > > index 203e26da5f92..e5cc8848dcb3 100644 > > > --- a/Documentation/admin-guide/mm/userfaultfd.rst > > > +++ b/Documentation/admin-guide/mm/userfaultfd.rst > > > @@ -113,6 +113,9 @@ events, except page fault notifications, may be generated: > > > areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the analogous feature indicating > > > support for shmem virtual memory areas. > > > > > > +- ``UFFD_FEATURE_MOVE`` indicates that the kernel supports moving an > > > + existing page contents from userspace. > > > + > > > The userland application should set the feature flags it intends to use > > > when invoking the ``UFFDIO_API`` ioctl, to request that those features be > > > enabled if supported. > > > diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c > > > index a7c6ef764e63..ac52e0f99a69 100644 > > > --- a/fs/userfaultfd.c > > > +++ b/fs/userfaultfd.c > > > @@ -2039,6 +2039,66 @@ static inline unsigned int uffd_ctx_features(__u64 user_features) > > > return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED; > > > } > > > > > > +static int userfaultfd_remap(struct userfaultfd_ctx *ctx, > > > + unsigned long arg) > > > > If we do want to rename from REMAP to MOVE, we'd better rename the > > functions too, as "remap" still exists all over the place.. > > Ok. I thought that since the current implementation only remaps and > never copies it would be correct to keep "remap" in these internal > names and change that later if we support copying. But I'm fine with > renaming them now to avoid confusion. Will do. "move", not "copy", btw. Not a big deal, take your preference at each place; "remap" sometimes can read better, maybe. Fundamentally, I think it's because both "remap" and "move" work in 99% cases. That's also why I think either name would work here. > > > > > > > +{ > > > + __s64 ret; > > > + struct uffdio_move uffdio_move; > > > + struct uffdio_move __user *user_uffdio_move; > > > + struct userfaultfd_wake_range range; > > > + > > > + user_uffdio_move = (struct uffdio_move __user *) arg; > > > + > > > + ret = -EAGAIN; > > > + if (atomic_read(&ctx->mmap_changing)) > > > + goto out; > > > > I didn't notice this before, but I think we need to re-check this after > > taking target mm's mmap read lock.. > > Ack. > > > > > maybe we'd want to pass in ctx* into remap_pages(), more reasoning below. > > Makes sense. > > > > > > + > > > + ret = -EFAULT; > > > + if (copy_from_user(&uffdio_move, user_uffdio_move, > > > + /* don't copy "remap" last field */ > > > > s/remap/move/ > > Ack. > > > > > > + sizeof(uffdio_move)-sizeof(__s64))) > > > + goto out; > > > + > > > + ret = validate_range(ctx->mm, uffdio_move.dst, uffdio_move.len); > > > + if (ret) > > > + goto out; > > > + > > > + ret = validate_range(current->mm, uffdio_move.src, uffdio_move.len); > > > + if (ret) > > > + goto out; > > > + > > > + ret = -EINVAL; > > > + if (uffdio_move.mode & ~(UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES| > > > + UFFDIO_MOVE_MODE_DONTWAKE)) > > > + goto out; > > > + > > > + if (mmget_not_zero(ctx->mm)) { > > > + ret = remap_pages(ctx->mm, current->mm, > > > + uffdio_move.dst, uffdio_move.src, > > > + uffdio_move.len, uffdio_move.mode); > > > + mmput(ctx->mm); > > > + } else { > > > + return -ESRCH; > > > + } > > > + > > > + if (unlikely(put_user(ret, &user_uffdio_move->move))) > > > + return -EFAULT; > > > + if (ret < 0) > > > + goto out; > > > + > > > + /* len == 0 would wake all */ > > > + BUG_ON(!ret); > > > + range.len = ret; > > > + if (!(uffdio_move.mode & UFFDIO_MOVE_MODE_DONTWAKE)) { > > > + range.start = uffdio_move.dst; > > > + wake_userfault(ctx, &range); > > > + } > > > + ret = range.len == uffdio_move.len ? 0 : -EAGAIN; > > > + > > > +out: > > > + return ret; > > > +} > > > + > > > /* > > > * userland asks for a certain API version and we return which bits > > > * and ioctl commands are implemented in this kernel for such API > > > @@ -2131,6 +2191,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd, > > > case UFFDIO_ZEROPAGE: > > > ret = userfaultfd_zeropage(ctx, arg); > > > break; > > > + case UFFDIO_MOVE: > > > + ret = userfaultfd_remap(ctx, arg); > > > + break; > > > case UFFDIO_WRITEPROTECT: > > > ret = userfaultfd_writeprotect(ctx, arg); > > > break; > > > diff --git a/include/linux/rmap.h b/include/linux/rmap.h > > > index b26fe858fd44..8034eda972e5 100644 > > > --- a/include/linux/rmap.h > > > +++ b/include/linux/rmap.h > > > @@ -121,6 +121,11 @@ static inline void anon_vma_lock_write(struct anon_vma *anon_vma) > > > down_write(&anon_vma->root->rwsem); > > > } > > > > > > +static inline int anon_vma_trylock_write(struct anon_vma *anon_vma) > > > +{ > > > + return down_write_trylock(&anon_vma->root->rwsem); > > > +} > > > + > > > static inline void anon_vma_unlock_write(struct anon_vma *anon_vma) > > > { > > > up_write(&anon_vma->root->rwsem); > > > diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h > > > index f2dc19f40d05..ce8d20b57e8c 100644 > > > --- a/include/linux/userfaultfd_k.h > > > +++ b/include/linux/userfaultfd_k.h > > > @@ -93,6 +93,18 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm, > > > extern long uffd_wp_range(struct vm_area_struct *vma, > > > unsigned long start, unsigned long len, bool enable_wp); > > > > > > +/* remap_pages */ > > > +void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2); > > > +void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2); > > > +ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > > + unsigned long dst_start, unsigned long src_start, > > > + unsigned long len, __u64 flags); > > > +int remap_pages_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > > + pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, > > > + struct vm_area_struct *dst_vma, > > > + struct vm_area_struct *src_vma, > > > + unsigned long dst_addr, unsigned long src_addr); > > > + > > > /* mm helpers */ > > > static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma, > > > struct vm_userfaultfd_ctx vm_ctx) > > > diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h > > > index 0dbc81015018..2841e4ea8f2c 100644 > > > --- a/include/uapi/linux/userfaultfd.h > > > +++ b/include/uapi/linux/userfaultfd.h > > > @@ -41,7 +41,8 @@ > > > UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \ > > > UFFD_FEATURE_WP_UNPOPULATED | \ > > > UFFD_FEATURE_POISON | \ > > > - UFFD_FEATURE_WP_ASYNC) > > > + UFFD_FEATURE_WP_ASYNC | \ > > > + UFFD_FEATURE_MOVE) > > > #define UFFD_API_IOCTLS \ > > > ((__u64)1 << _UFFDIO_REGISTER | \ > > > (__u64)1 << _UFFDIO_UNREGISTER | \ > > > @@ -50,6 +51,7 @@ > > > ((__u64)1 << _UFFDIO_WAKE | \ > > > (__u64)1 << _UFFDIO_COPY | \ > > > (__u64)1 << _UFFDIO_ZEROPAGE | \ > > > + (__u64)1 << _UFFDIO_MOVE | \ > > > (__u64)1 << _UFFDIO_WRITEPROTECT | \ > > > (__u64)1 << _UFFDIO_CONTINUE | \ > > > (__u64)1 << _UFFDIO_POISON) > > > @@ -73,6 +75,7 @@ > > > #define _UFFDIO_WAKE (0x02) > > > #define _UFFDIO_COPY (0x03) > > > #define _UFFDIO_ZEROPAGE (0x04) > > > +#define _UFFDIO_MOVE (0x05) > > > #define _UFFDIO_WRITEPROTECT (0x06) > > > #define _UFFDIO_CONTINUE (0x07) > > > #define _UFFDIO_POISON (0x08) > > > @@ -92,6 +95,8 @@ > > > struct uffdio_copy) > > > #define UFFDIO_ZEROPAGE _IOWR(UFFDIO, _UFFDIO_ZEROPAGE, \ > > > struct uffdio_zeropage) > > > +#define UFFDIO_MOVE _IOWR(UFFDIO, _UFFDIO_MOVE, \ > > > + struct uffdio_move) > > > #define UFFDIO_WRITEPROTECT _IOWR(UFFDIO, _UFFDIO_WRITEPROTECT, \ > > > struct uffdio_writeprotect) > > > #define UFFDIO_CONTINUE _IOWR(UFFDIO, _UFFDIO_CONTINUE, \ > > > @@ -222,6 +227,9 @@ struct uffdio_api { > > > * asynchronous mode is supported in which the write fault is > > > * automatically resolved and write-protection is un-set. > > > * It implies UFFD_FEATURE_WP_UNPOPULATED. > > > + * > > > + * UFFD_FEATURE_MOVE indicates that the kernel supports moving an > > > + * existing page contents from userspace. > > > */ > > > #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) > > > #define UFFD_FEATURE_EVENT_FORK (1<<1) > > > @@ -239,6 +247,7 @@ struct uffdio_api { > > > #define UFFD_FEATURE_WP_UNPOPULATED (1<<13) > > > #define UFFD_FEATURE_POISON (1<<14) > > > #define UFFD_FEATURE_WP_ASYNC (1<<15) > > > +#define UFFD_FEATURE_MOVE (1<<16) > > > __u64 features; > > > > > > __u64 ioctls; > > > @@ -347,6 +356,24 @@ struct uffdio_poison { > > > __s64 updated; > > > }; > > > > > > +struct uffdio_move { > > > + __u64 dst; > > > + __u64 src; > > > + __u64 len; > > > + /* > > > + * Especially if used to atomically remove memory from the > > > + * address space the wake on the dst range is not needed. > > > + */ > > > +#define UFFDIO_MOVE_MODE_DONTWAKE ((__u64)1<<0) > > > +#define UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES ((__u64)1<<1) > > > + __u64 mode; > > > + /* > > > + * "move" is written by the ioctl and must be at the end: the > > > + * copy_from_user will not read the last 8 bytes. > > > + */ > > > + __s64 move; > > > +}; > > > + > > > /* > > > * Flags for the userfaultfd(2) system call itself. > > > */ > > > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > > > index 9656be95a542..6fac5c3d66e6 100644 > > > --- a/mm/huge_memory.c > > > +++ b/mm/huge_memory.c > > > @@ -2086,6 +2086,144 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, > > > return ret; > > > } > > > > > > +#ifdef CONFIG_USERFAULTFD > > > +/* > > > + * The PT lock for src_pmd and the mmap_lock for reading are held by > > > + * the caller, but it must return after releasing the > > > + * page_table_lock. Just move the page from src_pmd to dst_pmd if possible. > > > + * Return zero if succeeded in moving the page, -EAGAIN if it needs to be > > > + * repeated by the caller, or other errors in case of failure. > > > + */ > > > +int remap_pages_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > > + pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, > > > + struct vm_area_struct *dst_vma, > > > + struct vm_area_struct *src_vma, > > > + unsigned long dst_addr, unsigned long src_addr) > > > +{ > > > + pmd_t _dst_pmd, src_pmdval; > > > + struct page *src_page; > > > + struct folio *src_folio; > > > + struct anon_vma *src_anon_vma; > > > + spinlock_t *src_ptl, *dst_ptl; > > > + pgtable_t src_pgtable, dst_pgtable; > > > + struct mmu_notifier_range range; > > > + int err = 0; > > > + > > > + src_pmdval = *src_pmd; > > > + src_ptl = pmd_lockptr(src_mm, src_pmd); > > > + > > > + lockdep_assert_held(src_ptl); > > > + mmap_assert_locked(src_mm); > > > + mmap_assert_locked(dst_mm); > > > + > > > + BUG_ON(!pmd_none(dst_pmdval)); > > > + BUG_ON(src_addr & ~HPAGE_PMD_MASK); > > > + BUG_ON(dst_addr & ~HPAGE_PMD_MASK); > > > + > > > + if (!pmd_trans_huge(src_pmdval)) { > > > + spin_unlock(src_ptl); > > > + if (is_pmd_migration_entry(src_pmdval)) { > > > + pmd_migration_entry_wait(src_mm, &src_pmdval); > > > + return -EAGAIN; > > > + } > > > + return -ENOENT; > > > + } > > > + > > > + src_page = pmd_page(src_pmdval); > > > + if (unlikely(!PageAnonExclusive(src_page))) { > > > + spin_unlock(src_ptl); > > > + return -EBUSY; > > > + } > > > + > > > + src_folio = page_folio(src_page); > > > + folio_get(src_folio); > > > + spin_unlock(src_ptl); > > > + > > > + /* preallocate dst_pgtable if needed */ > > > + if (dst_mm != src_mm) { > > > + dst_pgtable = pte_alloc_one(dst_mm); > > > + if (unlikely(!dst_pgtable)) { > > > + err = -ENOMEM; > > > + goto put_folio; > > > + } > > > + } else { > > > + dst_pgtable = NULL; > > > + } > > > + > > > > IIUC Lokesh's comment applies here, we probably need the > > flush_cache_range(), not for x86 but for the other ones.. > > > > cachetlb.rst: > > > > Next, we have the cache flushing interfaces. In general, when Linux > > is changing an existing virtual-->physical mapping to a new value, > > the sequence will be in one of the following forms:: > > > > 1) flush_cache_mm(mm); > > change_all_page_tables_of(mm); > > flush_tlb_mm(mm); > > > > 2) flush_cache_range(vma, start, end); > > change_range_of_page_tables(mm, start, end); > > flush_tlb_range(vma, start, end); > > > > 3) flush_cache_page(vma, addr, pfn); > > set_pte(pte_pointer, new_pte_val); > > flush_tlb_page(vma, addr); > > Thanks for the reference. I guess that's to support VIVT caches? I'm not 100% sure VIVT the only case, but.. yeah flush anything to ram as long as things cached in va form would be required. > > > > > > + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr, > > > + src_addr + HPAGE_PMD_SIZE); > > > + mmu_notifier_invalidate_range_start(&range); > > > + > > > + folio_lock(src_folio); > > > + > > > + /* > > > + * split_huge_page walks the anon_vma chain without the page > > > + * lock. Serialize against it with the anon_vma lock, the page > > > + * lock is not enough. > > > + */ > > > + src_anon_vma = folio_get_anon_vma(src_folio); > > > + if (!src_anon_vma) { > > > + err = -EAGAIN; > > > + goto unlock_folio; > > > + } > > > + anon_vma_lock_write(src_anon_vma); > > > + > > > + dst_ptl = pmd_lockptr(dst_mm, dst_pmd); > > > + double_pt_lock(src_ptl, dst_ptl); > > > + if (unlikely(!pmd_same(*src_pmd, src_pmdval) || > > > + !pmd_same(*dst_pmd, dst_pmdval))) { > > > + double_pt_unlock(src_ptl, dst_ptl); > > > + err = -EAGAIN; > > > + goto put_anon_vma; > > > + } > > > + if (!PageAnonExclusive(&src_folio->page)) { > > > + double_pt_unlock(src_ptl, dst_ptl); > > > + err = -EBUSY; > > > + goto put_anon_vma; > > > + } > > > + > > > + BUG_ON(!folio_test_head(src_folio)); > > > + BUG_ON(!folio_test_anon(src_folio)); > > > + > > > + folio_move_anon_rmap(src_folio, dst_vma); > > > + WRITE_ONCE(src_folio->index, linear_page_index(dst_vma, dst_addr)); > > > + > > > + src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd); > > > + _dst_pmd = mk_huge_pmd(&src_folio->page, dst_vma->vm_page_prot); > > > + _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma); > > > > Last time the conclusion is we leverage can_change_pmd_writable(), no? > > After your explanation that this works correctly for soft-dirty and > UFFD_WP I thought the only thing left to handle was the check for > VM_WRITE in both src_vma and dst_vma (which I added into > validate_remap_areas()). Maybe I misunderstood and if so, I can > replace the above PageAnonExclusive() with can_change_pmd_writable() > (note that we err out on VM_SHARED VMAs, so PageAnonExclusive() will > be included in that check). I think we still need PageAnonExclusive() because that's the first guard to decide whether the page can be moved over at all. What I meant is something like keeping that, then: if (pmd_soft_dirty(src_pmdval)) _dst_pmd = pmd_swp_mksoft_dirty(_dst_pmd); if (pmd_uffd_wp(src_pmdval)) _dst_pmd = pte_swp_mkuffd_wp(swp_pte); if (can_change_pmd_writable(_dst_vma, addr, _dst_pmd)) _dst_pmd = pmd_mkwrite(_dst_pmd, dst_vma); But I'm not really sure anyone can leverage that, especially after I just saw move_soft_dirty_pte(): mremap() treat everything dirty after movement. I don't think there's a clear definition of how we treat memory dirty after remap. Maybe we should follow what it does with mremap()? Then your current code is fine. Maybe that's the better start. > > > > > > + set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd); > > > + > > > + src_pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd); > > > + if (dst_pgtable) { > > > + pgtable_trans_huge_deposit(dst_mm, dst_pmd, dst_pgtable); > > > + pte_free(src_mm, src_pgtable); > > > + dst_pgtable = NULL; > > > + > > > + mm_inc_nr_ptes(dst_mm); > > > + mm_dec_nr_ptes(src_mm); > > > + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); > > > + add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR); > > > + } else { > > > + pgtable_trans_huge_deposit(dst_mm, dst_pmd, src_pgtable); > > > + } > > > + double_pt_unlock(src_ptl, dst_ptl); > > > + > > > +put_anon_vma: > > > + anon_vma_unlock_write(src_anon_vma); > > > + put_anon_vma(src_anon_vma); > > > +unlock_folio: > > > + /* unblock rmap walks */ > > > + folio_unlock(src_folio); > > > + mmu_notifier_invalidate_range_end(&range); > > > + if (dst_pgtable) > > > + pte_free(dst_mm, dst_pgtable); > > > +put_folio: > > > + folio_put(src_folio); > > > + > > > + return err; > > > +} > > > +#endif /* CONFIG_USERFAULTFD */ > > > + > > > /* > > > * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise. > > > * > > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > > > index 2b5c0321d96b..0c1ee7172852 100644 > > > --- a/mm/khugepaged.c > > > +++ b/mm/khugepaged.c > > > @@ -1136,6 +1136,9 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, > > > * Prevent all access to pagetables with the exception of > > > * gup_fast later handled by the ptep_clear_flush and the VM > > > * handled by the anon_vma lock + PG_lock. > > > + * > > > + * UFFDIO_MOVE is prevented to race as well thanks to the > > > + * mmap_lock. > > > */ > > > mmap_write_lock(mm); > > > result = hugepage_vma_revalidate(mm, address, true, &vma, cc); > > > diff --git a/mm/rmap.c b/mm/rmap.c > > > index f9ddc50269d2..a5919cac9a08 100644 > > > --- a/mm/rmap.c > > > +++ b/mm/rmap.c > > > @@ -490,6 +490,12 @@ void __init anon_vma_init(void) > > > * page_remove_rmap() that the anon_vma pointer from page->mapping is valid > > > * if there is a mapcount, we can dereference the anon_vma after observing > > > * those. > > > + * > > > + * NOTE: the caller should normally hold folio lock when calling this. If > > > + * not, the caller needs to double check the anon_vma didn't change after > > > + * taking the anon_vma lock for either read or write (UFFDIO_MOVE can modify it > > > + * concurrently without folio lock protection). See folio_lock_anon_vma_read() > > > + * which has already covered that, and comment above remap_pages(). > > > */ > > > struct anon_vma *folio_get_anon_vma(struct folio *folio) > > > { > > > diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c > > > index 96d9eae5c7cc..45ce1a8b8ab9 100644 > > > --- a/mm/userfaultfd.c > > > +++ b/mm/userfaultfd.c > > > @@ -842,3 +842,605 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, > > > mmap_read_unlock(dst_mm); > > > return err; > > > } > > > + > > > + > > > +void double_pt_lock(spinlock_t *ptl1, > > > + spinlock_t *ptl2) > > > + __acquires(ptl1) > > > + __acquires(ptl2) > > > +{ > > > + spinlock_t *ptl_tmp; > > > + > > > + if (ptl1 > ptl2) { > > > + /* exchange ptl1 and ptl2 */ > > > + ptl_tmp = ptl1; > > > + ptl1 = ptl2; > > > + ptl2 = ptl_tmp; > > > + } > > > + /* lock in virtual address order to avoid lock inversion */ > > > + spin_lock(ptl1); > > > + if (ptl1 != ptl2) > > > + spin_lock_nested(ptl2, SINGLE_DEPTH_NESTING); > > > + else > > > + __acquire(ptl2); > > > +} > > > + > > > +void double_pt_unlock(spinlock_t *ptl1, > > > + spinlock_t *ptl2) > > > + __releases(ptl1) > > > + __releases(ptl2) > > > +{ > > > + spin_unlock(ptl1); > > > + if (ptl1 != ptl2) > > > + spin_unlock(ptl2); > > > + else > > > + __release(ptl2); > > > +} > > > + > > > + > > > +static int remap_present_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > > + struct vm_area_struct *dst_vma, > > > + struct vm_area_struct *src_vma, > > > + unsigned long dst_addr, unsigned long src_addr, > > > + pte_t *dst_pte, pte_t *src_pte, > > > + pte_t orig_dst_pte, pte_t orig_src_pte, > > > + spinlock_t *dst_ptl, spinlock_t *src_ptl, > > > + struct folio *src_folio) > > > +{ > > > + double_pt_lock(dst_ptl, src_ptl); > > > + > > > + if (!pte_same(*src_pte, orig_src_pte) || > > > + !pte_same(*dst_pte, orig_dst_pte)) { > > > + double_pt_unlock(dst_ptl, src_ptl); > > > + return -EAGAIN; > > > + } > > > + if (folio_test_large(src_folio) || > > > + !PageAnonExclusive(&src_folio->page)) { > > > + double_pt_unlock(dst_ptl, src_ptl); > > > + return -EBUSY; > > > + } > > > + > > > + BUG_ON(!folio_test_anon(src_folio)); > > > + > > > + folio_move_anon_rmap(src_folio, dst_vma); > > > + WRITE_ONCE(src_folio->index, linear_page_index(dst_vma, dst_addr)); > > > + > > > + orig_src_pte = ptep_clear_flush(src_vma, src_addr, src_pte); > > > + orig_dst_pte = mk_pte(&src_folio->page, dst_vma->vm_page_prot); > > > + orig_dst_pte = maybe_mkwrite(pte_mkdirty(orig_dst_pte), dst_vma); > > > > can_change_pte_writable()? > > Same as my previous comment. If that's still needed I'll replace the > above PageAnonExclusive() check with can_change_pte_writable(). If no one else sees any problem, let's keep your current code, per my above observations.. to match mremap(), also keep it simple. One more thing I just remembered on memcg: only uncharge+charge may not work, I think the lruvec needs to be maintained as well, or memcg shrink can try to swap some irrelevant page at least, and memcg accounting can also go wrong. AFAICT, that means something like another pair of: folio_isolate_lru() + folio_putback_lru() Besides the charge/uncharge. Yu Zhao should be familiar with that code, maybe you can double check with him before sending the new version. I think this will belong to the separate patch to add cross-mm support, but please also double check even just in case there can be implication of single-mm that I missed. Please also don't feel stressed over cross-mm support: at some point if you see that separate patch grows we can stop from there, listing all the cross-mm todos/investigations in the cover letter and start with single-mm. Thanks,
On 22.10.23 17:46, Peter Xu wrote: > On Fri, Oct 20, 2023 at 07:16:19PM +0200, David Hildenbrand wrote: >> These are rather the vibes I'm getting from Peter. "Why rename it, could >> confuse people because the original patches are old", "Why exclude it if it >> has been included in the original patches". Not the kind of reasoning I can >> relate to when it comes to upstreaming some patches. > > You can't blame anyone if you misunderstood and biased the question. > > The first question is definitely valid, even until now. You guys still > prefer to rename it, which I'm totally fine with. > > The 2nd question is wrong from your interpretation. That's not my point, > at least not starting from a few replies already. What I was asking for is > why such page movement between mm is dangerous. I don't think I get solid > answers even until now. > > Noticing "memcg is missing" is not an argument for "cross-mm is dangerous", > it's a review comment. Suren can address that. > > You'll propose a new feature that may tag an mm is not an argument either, > if it's not merged yet. We can also address that depending on what it is, > also on which lands earlier. > > It'll be good to discuss these details even in a single-mm support. Anyone > would like to add that can already refer to discussion in this thread. > > I hope I'm clear. > I said everything I had to say, go read what I wrote.
Focusing on validate_remap_areas(): > + > +static int validate_remap_areas(struct vm_area_struct *src_vma, > + struct vm_area_struct *dst_vma) > +{ > + /* Only allow remapping if both have the same access and protection */ > + if ((src_vma->vm_flags & VM_ACCESS_FLAGS) != (dst_vma->vm_flags & VM_ACCESS_FLAGS) || > + pgprot_val(src_vma->vm_page_prot) != pgprot_val(dst_vma->vm_page_prot)) > + return -EINVAL; Makes sense. I do wonder about pkey and friends and if we even have to so anything special. > + > + /* Only allow remapping if both are mlocked or both aren't */ > + if ((src_vma->vm_flags & VM_LOCKED) != (dst_vma->vm_flags & VM_LOCKED)) > + return -EINVAL; > + > + if (!(src_vma->vm_flags & VM_WRITE) || !(dst_vma->vm_flags & VM_WRITE)) > + return -EINVAL; Why does one of both need VM_WRITE? If one really needs it, then the destination (where we're moving stuff to). > + > + /* > + * Be strict and only allow remap_pages if either the src or > + * dst range is registered in the userfaultfd to prevent > + * userland errors going unnoticed. As far as the VM > + * consistency is concerned, it would be perfectly safe to > + * remove this check, but there's no useful usage for > + * remap_pages ouside of userfaultfd registered ranges. This > + * is after all why it is an ioctl belonging to the > + * userfaultfd and not a syscall. I think the last sentence is the important bit and the comment can likely be reduced. > + * > + * Allow both vmas to be registered in the userfaultfd, just > + * in case somebody finds a way to make such a case useful. > + * Normally only one of the two vmas would be registered in > + * the userfaultfd. Should we just check the destination? That makes most sense to me, because with uffd we are resolving uffd-events. And just like copy/zeropage we want to resolve a page fault ("userfault") of a non-present page on the destination. > + */ > + if (!dst_vma->vm_userfaultfd_ctx.ctx && > + !src_vma->vm_userfaultfd_ctx.ctx) > + return -EINVAL; > + > + /* > + * FIXME: only allow remapping across anonymous vmas, > + * tmpfs should be added. > + */ > + if (!vma_is_anonymous(src_vma) || !vma_is_anonymous(dst_vma)) > + return -EINVAL; Why a FIXME here? Just drop the comment completely or replace it with "We only allow to remap anonymous folios accross anonymous VMAs". > + > + /* > + * Ensure the dst_vma has a anon_vma or this page > + * would get a NULL anon_vma when moved in the > + * dst_vma. > + */ > + if (unlikely(anon_vma_prepare(dst_vma))) > + return -ENOMEM; Makes sense. > + > + return 0; > +}
On 23.10.23 14:29, David Hildenbrand wrote: >> + >> + /* Only allow remapping if both are mlocked or both aren't */ >> + if ((src_vma->vm_flags & VM_LOCKED) != (dst_vma->vm_flags & VM_LOCKED)) >> + return -EINVAL; >> + >> + if (!(src_vma->vm_flags & VM_WRITE) || !(dst_vma->vm_flags & VM_WRITE)) >> + return -EINVAL; > > Why does one of both need VM_WRITE? If one really needs it, then the > destination (where we're moving stuff to). Just realized that we want both to be writable. If you have this in place, there is no need to use maybe*_mkwrite(), you can use the non-maybe variants. I recall that for UFFDIO_COPY we even support PROT_NONE VMAs, is there any reason why we want to have different semantics here?
On 23.10.23 14:03, David Hildenbrand wrote: > On 22.10.23 17:46, Peter Xu wrote: >> On Fri, Oct 20, 2023 at 07:16:19PM +0200, David Hildenbrand wrote: >>> These are rather the vibes I'm getting from Peter. "Why rename it, could >>> confuse people because the original patches are old", "Why exclude it if it >>> has been included in the original patches". Not the kind of reasoning I can >>> relate to when it comes to upstreaming some patches. >> >> You can't blame anyone if you misunderstood and biased the question. >> >> The first question is definitely valid, even until now. You guys still >> prefer to rename it, which I'm totally fine with. >> >> The 2nd question is wrong from your interpretation. That's not my point, >> at least not starting from a few replies already. What I was asking for is >> why such page movement between mm is dangerous. I don't think I get solid >> answers even until now. >> >> Noticing "memcg is missing" is not an argument for "cross-mm is dangerous", >> it's a review comment. Suren can address that. >> >> You'll propose a new feature that may tag an mm is not an argument either, >> if it's not merged yet. We can also address that depending on what it is, >> also on which lands earlier. >> >> It'll be good to discuss these details even in a single-mm support. Anyone >> would like to add that can already refer to discussion in this thread. >> >> I hope I'm clear. >> > > I said everything I had to say, go read what I wrote. Re-read your message after flying over first couple of paragraphs previously a bit quick too quickly (can easily happen when I'm told that I misunderstand questions and read them in a "biased" way). I'll happy to discuss cross-mm support once we actually need it. I just don't see the need to spend any energy on that right now, without any users on the horizon. [(a) I didn't blame anybody, I said that I don't understand the reasoning. (b) I hope I made it clear that this is added complexity (and not just currently dangerous) and so far I haven't heard a compelling argument why we should do any of that or even spend our time discussing that. (c) I never used "memcg is missing" as an argument for "cross-mm is dangerous", all about added complexity without actual users. (d) "it easily shows that there are cases where this will require extra work -- without any current benefits" -- is IMHO a perfectly fine argument against complexity that currently nobody needs]
On Mon, Oct 23, 2023 at 9:36 AM David Hildenbrand <david@redhat.com> wrote: > > On 23.10.23 14:03, David Hildenbrand wrote: > > On 22.10.23 17:46, Peter Xu wrote: > >> On Fri, Oct 20, 2023 at 07:16:19PM +0200, David Hildenbrand wrote: > >>> These are rather the vibes I'm getting from Peter. "Why rename it, could > >>> confuse people because the original patches are old", "Why exclude it if it > >>> has been included in the original patches". Not the kind of reasoning I can > >>> relate to when it comes to upstreaming some patches. > >> > >> You can't blame anyone if you misunderstood and biased the question. > >> > >> The first question is definitely valid, even until now. You guys still > >> prefer to rename it, which I'm totally fine with. > >> > >> The 2nd question is wrong from your interpretation. That's not my point, > >> at least not starting from a few replies already. What I was asking for is > >> why such page movement between mm is dangerous. I don't think I get solid > >> answers even until now. > >> > >> Noticing "memcg is missing" is not an argument for "cross-mm is dangerous", > >> it's a review comment. Suren can address that. > >> > >> You'll propose a new feature that may tag an mm is not an argument either, > >> if it's not merged yet. We can also address that depending on what it is, > >> also on which lands earlier. > >> > >> It'll be good to discuss these details even in a single-mm support. Anyone > >> would like to add that can already refer to discussion in this thread. > >> > >> I hope I'm clear. > >> > > > > I said everything I had to say, go read what I wrote. > > Re-read your message after flying over first couple of paragraphs > previously a bit quick too quickly (can easily happen when I'm told that > I misunderstand questions and read them in a "biased" way). > > I'll happy to discuss cross-mm support once we actually need it. I just > don't see the need to spend any energy on that right now, without any > users on the horizon. > > [(a) I didn't blame anybody, I said that I don't understand the > reasoning. (b) I hope I made it clear that this is added complexity (and > not just currently dangerous) and so far I haven't heard a compelling > argument why we should do any of that or even spend our time discussing > that. (c) I never used "memcg is missing" as an argument for "cross-mm > is dangerous", all about added complexity without actual users. (d) "it > easily shows that there are cases where this will require extra work -- > without any current benefits" -- is IMHO a perfectly fine argument > against complexity that currently nobody needs] Thanks for the discussion, folks! I think posting the single-mm first and then following up with cross-mm and its test would help us move forward. That will provide functionality that is needed today quickly without unnecessary distractions and will give us more time to discuss cross-mm support. Also we will be able to test single-mm in isolation and make it more solid before moving onto cross-mm. I'll try to post the next version sometime this week. Thanks, Suren. > > -- > Cheers, > > David / dhildenb >
On Sun, Oct 22, 2023 at 10:02 AM Peter Xu <peterx@redhat.com> wrote: > > On Thu, Oct 19, 2023 at 02:24:06PM -0700, Suren Baghdasaryan wrote: > > On Thu, Oct 12, 2023 at 3:00 PM Peter Xu <peterx@redhat.com> wrote: > > > > > > On Sun, Oct 08, 2023 at 11:42:27PM -0700, Suren Baghdasaryan wrote: > > > > From: Andrea Arcangeli <aarcange@redhat.com> > > > > > > > > Implement the uABI of UFFDIO_MOVE ioctl. > > > > UFFDIO_COPY performs ~20% better than UFFDIO_MOVE when the application > > > > needs pages to be allocated [1]. However, with UFFDIO_MOVE, if pages are > > > > available (in userspace) for recycling, as is usually the case in heap > > > > compaction algorithms, then we can avoid the page allocation and memcpy > > > > (done by UFFDIO_COPY). Also, since the pages are recycled in the > > > > userspace, we avoid the need to release (via madvise) the pages back to > > > > the kernel [2]. > > > > We see over 40% reduction (on a Google pixel 6 device) in the compacting > > > > thread’s completion time by using UFFDIO_MOVE vs. UFFDIO_COPY. This was > > > > measured using a benchmark that emulates a heap compaction implementation > > > > using userfaultfd (to allow concurrent accesses by application threads). > > > > More details of the usecase are explained in [2]. > > > > Furthermore, UFFDIO_MOVE enables moving swapped-out pages without > > > > touching them within the same vma. Today, it can only be done by mremap, > > > > however it forces splitting the vma. > > > > > > > > [1] https://lore.kernel.org/all/1425575884-2574-1-git-send-email-aarcange@redhat.com/ > > > > [2] https://lore.kernel.org/linux-mm/CA+EESO4uO84SSnBhArH4HvLNhaUQ5nZKNKXqxRCyjniNVjp0Aw@mail.gmail.com/ > > > > > > > > Update for the ioctl_userfaultfd(2) manpage: > > > > > > > > UFFDIO_MOVE > > > > (Since Linux xxx) Move a continuous memory chunk into the > > > > userfault registered range and optionally wake up the blocked > > > > thread. The source and destination addresses and the number of > > > > bytes to move are specified by the src, dst, and len fields of > > > > the uffdio_move structure pointed to by argp: > > > > > > > > struct uffdio_move { > > > > __u64 dst; /* Destination of move */ > > > > __u64 src; /* Source of move */ > > > > __u64 len; /* Number of bytes to move */ > > > > __u64 mode; /* Flags controlling behavior of move */ > > > > __s64 move; /* Number of bytes moved, or negated error */ > > > > }; > > > > > > > > The following value may be bitwise ORed in mode to change the > > > > behavior of the UFFDIO_MOVE operation: > > > > > > > > UFFDIO_MOVE_MODE_DONTWAKE > > > > Do not wake up the thread that waits for page-fault > > > > resolution > > > > > > > > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES > > > > Allow holes in the source virtual range that is being moved. > > > > When not specified, the holes will result in ENOENT error. > > > > When specified, the holes will be accounted as successfully > > > > moved memory. This is mostly useful to move hugepage aligned > > > > virtual regions without knowing if there are transparent > > > > hugepages in the regions or not, but preventing the risk of > > > > having to split the hugepage during the operation. > > > > > > > > The move field is used by the kernel to return the number of > > > > bytes that was actually moved, or an error (a negated errno- > > > > style value). If the value returned in move doesn't match the > > > > value that was specified in len, the operation fails with the > > > > error EAGAIN. The move field is output-only; it is not read by > > > > the UFFDIO_MOVE operation. > > > > > > > > The operation may fail for various reasons. Usually, remapping of > > > > pages that are not exclusive to the given process fail; once KSM > > > > might deduplicate pages or fork() COW-shares pages during fork() > > > > with child processes, they are no longer exclusive. Further, the > > > > kernel might only perform lightweight checks for detecting whether > > > > the pages are exclusive, and return -EBUSY in case that check fails. > > > > To make the operation more likely to succeed, KSM should be > > > > disabled, fork() should be avoided or MADV_DONTFORK should be > > > > configured for the source VMA before fork(). > > > > > > > > This ioctl(2) operation returns 0 on success. In this case, the > > > > entire area was moved. On error, -1 is returned and errno is > > > > set to indicate the error. Possible errors include: > > > > > > > > EAGAIN The number of bytes moved (i.e., the value returned in > > > > the move field) does not equal the value that was > > > > specified in the len field. > > > > > > > > EINVAL Either dst or len was not a multiple of the system page > > > > size, or the range specified by src and len or dst and len > > > > was invalid. > > > > > > > > EINVAL An invalid bit was specified in the mode field. > > > > > > > > ENOENT > > > > The source virtual memory range has unmapped holes and > > > > UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES is not set. > > > > > > > > EEXIST > > > > The destination virtual memory range is fully or partially > > > > mapped. > > > > > > > > EBUSY > > > > The pages in the source virtual memory range are not > > > > exclusive to the process. The kernel might only perform > > > > lightweight checks for detecting whether the pages are > > > > exclusive. To make the operation more likely to succeed, > > > > KSM should be disabled, fork() should be avoided or > > > > MADV_DONTFORK should be configured for the source virtual > > > > memory area before fork(). > > > > > > > > ENOMEM Allocating memory needed for the operation failed. > > > > > > > > ESRCH > > > > The faulting process has exited at the time of a > > > > > > Nit pick comment for future man page: there's no faulting process in this > > > context. Perhaps "target process"? > > > > Ack. > > > > > > > > > UFFDIO_MOVE operation. > > > > > > > > Signed-off-by: Andrea Arcangeli <aarcange@redhat.com> > > > > Signed-off-by: Suren Baghdasaryan <surenb@google.com> > > > > --- > > > > Documentation/admin-guide/mm/userfaultfd.rst | 3 + > > > > fs/userfaultfd.c | 63 ++ > > > > include/linux/rmap.h | 5 + > > > > include/linux/userfaultfd_k.h | 12 + > > > > include/uapi/linux/userfaultfd.h | 29 +- > > > > mm/huge_memory.c | 138 +++++ > > > > mm/khugepaged.c | 3 + > > > > mm/rmap.c | 6 + > > > > mm/userfaultfd.c | 602 +++++++++++++++++++ > > > > 9 files changed, 860 insertions(+), 1 deletion(-) > > > > > > > > diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst > > > > index 203e26da5f92..e5cc8848dcb3 100644 > > > > --- a/Documentation/admin-guide/mm/userfaultfd.rst > > > > +++ b/Documentation/admin-guide/mm/userfaultfd.rst > > > > @@ -113,6 +113,9 @@ events, except page fault notifications, may be generated: > > > > areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the analogous feature indicating > > > > support for shmem virtual memory areas. > > > > > > > > +- ``UFFD_FEATURE_MOVE`` indicates that the kernel supports moving an > > > > + existing page contents from userspace. > > > > + > > > > The userland application should set the feature flags it intends to use > > > > when invoking the ``UFFDIO_API`` ioctl, to request that those features be > > > > enabled if supported. > > > > diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c > > > > index a7c6ef764e63..ac52e0f99a69 100644 > > > > --- a/fs/userfaultfd.c > > > > +++ b/fs/userfaultfd.c > > > > @@ -2039,6 +2039,66 @@ static inline unsigned int uffd_ctx_features(__u64 user_features) > > > > return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED; > > > > } > > > > > > > > +static int userfaultfd_remap(struct userfaultfd_ctx *ctx, > > > > + unsigned long arg) > > > > > > If we do want to rename from REMAP to MOVE, we'd better rename the > > > functions too, as "remap" still exists all over the place.. > > > > Ok. I thought that since the current implementation only remaps and > > never copies it would be correct to keep "remap" in these internal > > names and change that later if we support copying. But I'm fine with > > renaming them now to avoid confusion. Will do. > > "move", not "copy", btw. > > Not a big deal, take your preference at each place; "remap" sometimes can > read better, maybe. Fundamentally, I think it's because both "remap" and > "move" work in 99% cases. That's also why I think either name would work > here. Ok, "move" it is. That will avoid unnecessary churn in the future if we decide to implement the copy fallback. > > > > > > > > > > > > +{ > > > > + __s64 ret; > > > > + struct uffdio_move uffdio_move; > > > > + struct uffdio_move __user *user_uffdio_move; > > > > + struct userfaultfd_wake_range range; > > > > + > > > > + user_uffdio_move = (struct uffdio_move __user *) arg; > > > > + > > > > + ret = -EAGAIN; > > > > + if (atomic_read(&ctx->mmap_changing)) > > > > + goto out; > > > > > > I didn't notice this before, but I think we need to re-check this after > > > taking target mm's mmap read lock.. > > > > Ack. > > > > > > > > maybe we'd want to pass in ctx* into remap_pages(), more reasoning below. > > > > Makes sense. > > > > > > > > > + > > > > + ret = -EFAULT; > > > > + if (copy_from_user(&uffdio_move, user_uffdio_move, > > > > + /* don't copy "remap" last field */ > > > > > > s/remap/move/ > > > > Ack. > > > > > > > > > + sizeof(uffdio_move)-sizeof(__s64))) > > > > + goto out; > > > > + > > > > + ret = validate_range(ctx->mm, uffdio_move.dst, uffdio_move.len); > > > > + if (ret) > > > > + goto out; > > > > + > > > > + ret = validate_range(current->mm, uffdio_move.src, uffdio_move.len); > > > > + if (ret) > > > > + goto out; > > > > + > > > > + ret = -EINVAL; > > > > + if (uffdio_move.mode & ~(UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES| > > > > + UFFDIO_MOVE_MODE_DONTWAKE)) > > > > + goto out; > > > > + > > > > + if (mmget_not_zero(ctx->mm)) { > > > > + ret = remap_pages(ctx->mm, current->mm, > > > > + uffdio_move.dst, uffdio_move.src, > > > > + uffdio_move.len, uffdio_move.mode); > > > > + mmput(ctx->mm); > > > > + } else { > > > > + return -ESRCH; > > > > + } > > > > + > > > > + if (unlikely(put_user(ret, &user_uffdio_move->move))) > > > > + return -EFAULT; > > > > + if (ret < 0) > > > > + goto out; > > > > + > > > > + /* len == 0 would wake all */ > > > > + BUG_ON(!ret); > > > > + range.len = ret; > > > > + if (!(uffdio_move.mode & UFFDIO_MOVE_MODE_DONTWAKE)) { > > > > + range.start = uffdio_move.dst; > > > > + wake_userfault(ctx, &range); > > > > + } > > > > + ret = range.len == uffdio_move.len ? 0 : -EAGAIN; > > > > + > > > > +out: > > > > + return ret; > > > > +} > > > > + > > > > /* > > > > * userland asks for a certain API version and we return which bits > > > > * and ioctl commands are implemented in this kernel for such API > > > > @@ -2131,6 +2191,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd, > > > > case UFFDIO_ZEROPAGE: > > > > ret = userfaultfd_zeropage(ctx, arg); > > > > break; > > > > + case UFFDIO_MOVE: > > > > + ret = userfaultfd_remap(ctx, arg); > > > > + break; > > > > case UFFDIO_WRITEPROTECT: > > > > ret = userfaultfd_writeprotect(ctx, arg); > > > > break; > > > > diff --git a/include/linux/rmap.h b/include/linux/rmap.h > > > > index b26fe858fd44..8034eda972e5 100644 > > > > --- a/include/linux/rmap.h > > > > +++ b/include/linux/rmap.h > > > > @@ -121,6 +121,11 @@ static inline void anon_vma_lock_write(struct anon_vma *anon_vma) > > > > down_write(&anon_vma->root->rwsem); > > > > } > > > > > > > > +static inline int anon_vma_trylock_write(struct anon_vma *anon_vma) > > > > +{ > > > > + return down_write_trylock(&anon_vma->root->rwsem); > > > > +} > > > > + > > > > static inline void anon_vma_unlock_write(struct anon_vma *anon_vma) > > > > { > > > > up_write(&anon_vma->root->rwsem); > > > > diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h > > > > index f2dc19f40d05..ce8d20b57e8c 100644 > > > > --- a/include/linux/userfaultfd_k.h > > > > +++ b/include/linux/userfaultfd_k.h > > > > @@ -93,6 +93,18 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm, > > > > extern long uffd_wp_range(struct vm_area_struct *vma, > > > > unsigned long start, unsigned long len, bool enable_wp); > > > > > > > > +/* remap_pages */ > > > > +void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2); > > > > +void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2); > > > > +ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > > > + unsigned long dst_start, unsigned long src_start, > > > > + unsigned long len, __u64 flags); > > > > +int remap_pages_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > > > + pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, > > > > + struct vm_area_struct *dst_vma, > > > > + struct vm_area_struct *src_vma, > > > > + unsigned long dst_addr, unsigned long src_addr); > > > > + > > > > /* mm helpers */ > > > > static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma, > > > > struct vm_userfaultfd_ctx vm_ctx) > > > > diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h > > > > index 0dbc81015018..2841e4ea8f2c 100644 > > > > --- a/include/uapi/linux/userfaultfd.h > > > > +++ b/include/uapi/linux/userfaultfd.h > > > > @@ -41,7 +41,8 @@ > > > > UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \ > > > > UFFD_FEATURE_WP_UNPOPULATED | \ > > > > UFFD_FEATURE_POISON | \ > > > > - UFFD_FEATURE_WP_ASYNC) > > > > + UFFD_FEATURE_WP_ASYNC | \ > > > > + UFFD_FEATURE_MOVE) > > > > #define UFFD_API_IOCTLS \ > > > > ((__u64)1 << _UFFDIO_REGISTER | \ > > > > (__u64)1 << _UFFDIO_UNREGISTER | \ > > > > @@ -50,6 +51,7 @@ > > > > ((__u64)1 << _UFFDIO_WAKE | \ > > > > (__u64)1 << _UFFDIO_COPY | \ > > > > (__u64)1 << _UFFDIO_ZEROPAGE | \ > > > > + (__u64)1 << _UFFDIO_MOVE | \ > > > > (__u64)1 << _UFFDIO_WRITEPROTECT | \ > > > > (__u64)1 << _UFFDIO_CONTINUE | \ > > > > (__u64)1 << _UFFDIO_POISON) > > > > @@ -73,6 +75,7 @@ > > > > #define _UFFDIO_WAKE (0x02) > > > > #define _UFFDIO_COPY (0x03) > > > > #define _UFFDIO_ZEROPAGE (0x04) > > > > +#define _UFFDIO_MOVE (0x05) > > > > #define _UFFDIO_WRITEPROTECT (0x06) > > > > #define _UFFDIO_CONTINUE (0x07) > > > > #define _UFFDIO_POISON (0x08) > > > > @@ -92,6 +95,8 @@ > > > > struct uffdio_copy) > > > > #define UFFDIO_ZEROPAGE _IOWR(UFFDIO, _UFFDIO_ZEROPAGE, \ > > > > struct uffdio_zeropage) > > > > +#define UFFDIO_MOVE _IOWR(UFFDIO, _UFFDIO_MOVE, \ > > > > + struct uffdio_move) > > > > #define UFFDIO_WRITEPROTECT _IOWR(UFFDIO, _UFFDIO_WRITEPROTECT, \ > > > > struct uffdio_writeprotect) > > > > #define UFFDIO_CONTINUE _IOWR(UFFDIO, _UFFDIO_CONTINUE, \ > > > > @@ -222,6 +227,9 @@ struct uffdio_api { > > > > * asynchronous mode is supported in which the write fault is > > > > * automatically resolved and write-protection is un-set. > > > > * It implies UFFD_FEATURE_WP_UNPOPULATED. > > > > + * > > > > + * UFFD_FEATURE_MOVE indicates that the kernel supports moving an > > > > + * existing page contents from userspace. > > > > */ > > > > #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) > > > > #define UFFD_FEATURE_EVENT_FORK (1<<1) > > > > @@ -239,6 +247,7 @@ struct uffdio_api { > > > > #define UFFD_FEATURE_WP_UNPOPULATED (1<<13) > > > > #define UFFD_FEATURE_POISON (1<<14) > > > > #define UFFD_FEATURE_WP_ASYNC (1<<15) > > > > +#define UFFD_FEATURE_MOVE (1<<16) > > > > __u64 features; > > > > > > > > __u64 ioctls; > > > > @@ -347,6 +356,24 @@ struct uffdio_poison { > > > > __s64 updated; > > > > }; > > > > > > > > +struct uffdio_move { > > > > + __u64 dst; > > > > + __u64 src; > > > > + __u64 len; > > > > + /* > > > > + * Especially if used to atomically remove memory from the > > > > + * address space the wake on the dst range is not needed. > > > > + */ > > > > +#define UFFDIO_MOVE_MODE_DONTWAKE ((__u64)1<<0) > > > > +#define UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES ((__u64)1<<1) > > > > + __u64 mode; > > > > + /* > > > > + * "move" is written by the ioctl and must be at the end: the > > > > + * copy_from_user will not read the last 8 bytes. > > > > + */ > > > > + __s64 move; > > > > +}; > > > > + > > > > /* > > > > * Flags for the userfaultfd(2) system call itself. > > > > */ > > > > diff --git a/mm/huge_memory.c b/mm/huge_memory.c > > > > index 9656be95a542..6fac5c3d66e6 100644 > > > > --- a/mm/huge_memory.c > > > > +++ b/mm/huge_memory.c > > > > @@ -2086,6 +2086,144 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, > > > > return ret; > > > > } > > > > > > > > +#ifdef CONFIG_USERFAULTFD > > > > +/* > > > > + * The PT lock for src_pmd and the mmap_lock for reading are held by > > > > + * the caller, but it must return after releasing the > > > > + * page_table_lock. Just move the page from src_pmd to dst_pmd if possible. > > > > + * Return zero if succeeded in moving the page, -EAGAIN if it needs to be > > > > + * repeated by the caller, or other errors in case of failure. > > > > + */ > > > > +int remap_pages_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > > > + pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, > > > > + struct vm_area_struct *dst_vma, > > > > + struct vm_area_struct *src_vma, > > > > + unsigned long dst_addr, unsigned long src_addr) > > > > +{ > > > > + pmd_t _dst_pmd, src_pmdval; > > > > + struct page *src_page; > > > > + struct folio *src_folio; > > > > + struct anon_vma *src_anon_vma; > > > > + spinlock_t *src_ptl, *dst_ptl; > > > > + pgtable_t src_pgtable, dst_pgtable; > > > > + struct mmu_notifier_range range; > > > > + int err = 0; > > > > + > > > > + src_pmdval = *src_pmd; > > > > + src_ptl = pmd_lockptr(src_mm, src_pmd); > > > > + > > > > + lockdep_assert_held(src_ptl); > > > > + mmap_assert_locked(src_mm); > > > > + mmap_assert_locked(dst_mm); > > > > + > > > > + BUG_ON(!pmd_none(dst_pmdval)); > > > > + BUG_ON(src_addr & ~HPAGE_PMD_MASK); > > > > + BUG_ON(dst_addr & ~HPAGE_PMD_MASK); > > > > + > > > > + if (!pmd_trans_huge(src_pmdval)) { > > > > + spin_unlock(src_ptl); > > > > + if (is_pmd_migration_entry(src_pmdval)) { > > > > + pmd_migration_entry_wait(src_mm, &src_pmdval); > > > > + return -EAGAIN; > > > > + } > > > > + return -ENOENT; > > > > + } > > > > + > > > > + src_page = pmd_page(src_pmdval); > > > > + if (unlikely(!PageAnonExclusive(src_page))) { > > > > + spin_unlock(src_ptl); > > > > + return -EBUSY; > > > > + } > > > > + > > > > + src_folio = page_folio(src_page); > > > > + folio_get(src_folio); > > > > + spin_unlock(src_ptl); > > > > + > > > > + /* preallocate dst_pgtable if needed */ > > > > + if (dst_mm != src_mm) { > > > > + dst_pgtable = pte_alloc_one(dst_mm); > > > > + if (unlikely(!dst_pgtable)) { > > > > + err = -ENOMEM; > > > > + goto put_folio; > > > > + } > > > > + } else { > > > > + dst_pgtable = NULL; > > > > + } > > > > + > > > > > > IIUC Lokesh's comment applies here, we probably need the > > > flush_cache_range(), not for x86 but for the other ones.. > > > > > > cachetlb.rst: > > > > > > Next, we have the cache flushing interfaces. In general, when Linux > > > is changing an existing virtual-->physical mapping to a new value, > > > the sequence will be in one of the following forms:: > > > > > > 1) flush_cache_mm(mm); > > > change_all_page_tables_of(mm); > > > flush_tlb_mm(mm); > > > > > > 2) flush_cache_range(vma, start, end); > > > change_range_of_page_tables(mm, start, end); > > > flush_tlb_range(vma, start, end); > > > > > > 3) flush_cache_page(vma, addr, pfn); > > > set_pte(pte_pointer, new_pte_val); > > > flush_tlb_page(vma, addr); > > > > Thanks for the reference. I guess that's to support VIVT caches? > > I'm not 100% sure VIVT the only case, but.. yeah flush anything to ram as > long as things cached in va form would be required. Ack. > > > > > > > > > > + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr, > > > > + src_addr + HPAGE_PMD_SIZE); > > > > + mmu_notifier_invalidate_range_start(&range); > > > > + > > > > + folio_lock(src_folio); > > > > + > > > > + /* > > > > + * split_huge_page walks the anon_vma chain without the page > > > > + * lock. Serialize against it with the anon_vma lock, the page > > > > + * lock is not enough. > > > > + */ > > > > + src_anon_vma = folio_get_anon_vma(src_folio); > > > > + if (!src_anon_vma) { > > > > + err = -EAGAIN; > > > > + goto unlock_folio; > > > > + } > > > > + anon_vma_lock_write(src_anon_vma); > > > > + > > > > + dst_ptl = pmd_lockptr(dst_mm, dst_pmd); > > > > + double_pt_lock(src_ptl, dst_ptl); > > > > + if (unlikely(!pmd_same(*src_pmd, src_pmdval) || > > > > + !pmd_same(*dst_pmd, dst_pmdval))) { > > > > + double_pt_unlock(src_ptl, dst_ptl); > > > > + err = -EAGAIN; > > > > + goto put_anon_vma; > > > > + } > > > > + if (!PageAnonExclusive(&src_folio->page)) { > > > > + double_pt_unlock(src_ptl, dst_ptl); > > > > + err = -EBUSY; > > > > + goto put_anon_vma; > > > > + } > > > > + > > > > + BUG_ON(!folio_test_head(src_folio)); > > > > + BUG_ON(!folio_test_anon(src_folio)); > > > > + > > > > + folio_move_anon_rmap(src_folio, dst_vma); > > > > + WRITE_ONCE(src_folio->index, linear_page_index(dst_vma, dst_addr)); > > > > + > > > > + src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd); > > > > + _dst_pmd = mk_huge_pmd(&src_folio->page, dst_vma->vm_page_prot); > > > > + _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma); > > > > > > Last time the conclusion is we leverage can_change_pmd_writable(), no? > > > > After your explanation that this works correctly for soft-dirty and > > UFFD_WP I thought the only thing left to handle was the check for > > VM_WRITE in both src_vma and dst_vma (which I added into > > validate_remap_areas()). Maybe I misunderstood and if so, I can > > replace the above PageAnonExclusive() with can_change_pmd_writable() > > (note that we err out on VM_SHARED VMAs, so PageAnonExclusive() will > > be included in that check). > > I think we still need PageAnonExclusive() because that's the first guard to > decide whether the page can be moved over at all. > > What I meant is something like keeping that, then: > > if (pmd_soft_dirty(src_pmdval)) > _dst_pmd = pmd_swp_mksoft_dirty(_dst_pmd); > if (pmd_uffd_wp(src_pmdval)) > _dst_pmd = pte_swp_mkuffd_wp(swp_pte); > if (can_change_pmd_writable(_dst_vma, addr, _dst_pmd)) > _dst_pmd = pmd_mkwrite(_dst_pmd, dst_vma); > > But I'm not really sure anyone can leverage that, especially after I just > saw move_soft_dirty_pte(): mremap() treat everything dirty after movement. > I don't think there's a clear definition of how we treat memory dirty after > remap. > > Maybe we should follow what it does with mremap()? Then your current code > is fine. Maybe that's the better start. I think that was the original intention, basically treating remapping as a write operation. Maybe I should add a comment here to make it more clear? > > > > > > > > > > + set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd); > > > > + > > > > + src_pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd); > > > > + if (dst_pgtable) { > > > > + pgtable_trans_huge_deposit(dst_mm, dst_pmd, dst_pgtable); > > > > + pte_free(src_mm, src_pgtable); > > > > + dst_pgtable = NULL; > > > > + > > > > + mm_inc_nr_ptes(dst_mm); > > > > + mm_dec_nr_ptes(src_mm); > > > > + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); > > > > + add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR); > > > > + } else { > > > > + pgtable_trans_huge_deposit(dst_mm, dst_pmd, src_pgtable); > > > > + } > > > > + double_pt_unlock(src_ptl, dst_ptl); > > > > + > > > > +put_anon_vma: > > > > + anon_vma_unlock_write(src_anon_vma); > > > > + put_anon_vma(src_anon_vma); > > > > +unlock_folio: > > > > + /* unblock rmap walks */ > > > > + folio_unlock(src_folio); > > > > + mmu_notifier_invalidate_range_end(&range); > > > > + if (dst_pgtable) > > > > + pte_free(dst_mm, dst_pgtable); > > > > +put_folio: > > > > + folio_put(src_folio); > > > > + > > > > + return err; > > > > +} > > > > +#endif /* CONFIG_USERFAULTFD */ > > > > + > > > > /* > > > > * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise. > > > > * > > > > diff --git a/mm/khugepaged.c b/mm/khugepaged.c > > > > index 2b5c0321d96b..0c1ee7172852 100644 > > > > --- a/mm/khugepaged.c > > > > +++ b/mm/khugepaged.c > > > > @@ -1136,6 +1136,9 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, > > > > * Prevent all access to pagetables with the exception of > > > > * gup_fast later handled by the ptep_clear_flush and the VM > > > > * handled by the anon_vma lock + PG_lock. > > > > + * > > > > + * UFFDIO_MOVE is prevented to race as well thanks to the > > > > + * mmap_lock. > > > > */ > > > > mmap_write_lock(mm); > > > > result = hugepage_vma_revalidate(mm, address, true, &vma, cc); > > > > diff --git a/mm/rmap.c b/mm/rmap.c > > > > index f9ddc50269d2..a5919cac9a08 100644 > > > > --- a/mm/rmap.c > > > > +++ b/mm/rmap.c > > > > @@ -490,6 +490,12 @@ void __init anon_vma_init(void) > > > > * page_remove_rmap() that the anon_vma pointer from page->mapping is valid > > > > * if there is a mapcount, we can dereference the anon_vma after observing > > > > * those. > > > > + * > > > > + * NOTE: the caller should normally hold folio lock when calling this. If > > > > + * not, the caller needs to double check the anon_vma didn't change after > > > > + * taking the anon_vma lock for either read or write (UFFDIO_MOVE can modify it > > > > + * concurrently without folio lock protection). See folio_lock_anon_vma_read() > > > > + * which has already covered that, and comment above remap_pages(). > > > > */ > > > > struct anon_vma *folio_get_anon_vma(struct folio *folio) > > > > { > > > > diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c > > > > index 96d9eae5c7cc..45ce1a8b8ab9 100644 > > > > --- a/mm/userfaultfd.c > > > > +++ b/mm/userfaultfd.c > > > > @@ -842,3 +842,605 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, > > > > mmap_read_unlock(dst_mm); > > > > return err; > > > > } > > > > + > > > > + > > > > +void double_pt_lock(spinlock_t *ptl1, > > > > + spinlock_t *ptl2) > > > > + __acquires(ptl1) > > > > + __acquires(ptl2) > > > > +{ > > > > + spinlock_t *ptl_tmp; > > > > + > > > > + if (ptl1 > ptl2) { > > > > + /* exchange ptl1 and ptl2 */ > > > > + ptl_tmp = ptl1; > > > > + ptl1 = ptl2; > > > > + ptl2 = ptl_tmp; > > > > + } > > > > + /* lock in virtual address order to avoid lock inversion */ > > > > + spin_lock(ptl1); > > > > + if (ptl1 != ptl2) > > > > + spin_lock_nested(ptl2, SINGLE_DEPTH_NESTING); > > > > + else > > > > + __acquire(ptl2); > > > > +} > > > > + > > > > +void double_pt_unlock(spinlock_t *ptl1, > > > > + spinlock_t *ptl2) > > > > + __releases(ptl1) > > > > + __releases(ptl2) > > > > +{ > > > > + spin_unlock(ptl1); > > > > + if (ptl1 != ptl2) > > > > + spin_unlock(ptl2); > > > > + else > > > > + __release(ptl2); > > > > +} > > > > + > > > > + > > > > +static int remap_present_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, > > > > + struct vm_area_struct *dst_vma, > > > > + struct vm_area_struct *src_vma, > > > > + unsigned long dst_addr, unsigned long src_addr, > > > > + pte_t *dst_pte, pte_t *src_pte, > > > > + pte_t orig_dst_pte, pte_t orig_src_pte, > > > > + spinlock_t *dst_ptl, spinlock_t *src_ptl, > > > > + struct folio *src_folio) > > > > +{ > > > > + double_pt_lock(dst_ptl, src_ptl); > > > > + > > > > + if (!pte_same(*src_pte, orig_src_pte) || > > > > + !pte_same(*dst_pte, orig_dst_pte)) { > > > > + double_pt_unlock(dst_ptl, src_ptl); > > > > + return -EAGAIN; > > > > + } > > > > + if (folio_test_large(src_folio) || > > > > + !PageAnonExclusive(&src_folio->page)) { > > > > + double_pt_unlock(dst_ptl, src_ptl); > > > > + return -EBUSY; > > > > + } > > > > + > > > > + BUG_ON(!folio_test_anon(src_folio)); > > > > + > > > > + folio_move_anon_rmap(src_folio, dst_vma); > > > > + WRITE_ONCE(src_folio->index, linear_page_index(dst_vma, dst_addr)); > > > > + > > > > + orig_src_pte = ptep_clear_flush(src_vma, src_addr, src_pte); > > > > + orig_dst_pte = mk_pte(&src_folio->page, dst_vma->vm_page_prot); > > > > + orig_dst_pte = maybe_mkwrite(pte_mkdirty(orig_dst_pte), dst_vma); > > > > > > can_change_pte_writable()? > > > > Same as my previous comment. If that's still needed I'll replace the > > above PageAnonExclusive() check with can_change_pte_writable(). > > If no one else sees any problem, let's keep your current code, per my above > observations.. to match mremap(), also keep it simple. Ack. > > One more thing I just remembered on memcg: only uncharge+charge may not > work, I think the lruvec needs to be maintained as well, or memcg shrink > can try to swap some irrelevant page at least, and memcg accounting can > also go wrong. > > AFAICT, that means something like another pair of: > > folio_isolate_lru() + folio_putback_lru() > > Besides the charge/uncharge. > > Yu Zhao should be familiar with that code, maybe you can double check with > him before sending the new version. Ok, I'll double check with Yu before sending cross-mm parts. > > I think this will belong to the separate patch to add cross-mm support, but > please also double check even just in case there can be implication of > single-mm that I missed. Hmm. For single-mm we do not recharge memcgs. Why would we need to isolate and put the pages back? Maybe I'm missing something... I'll ask Yu in any case. > > Please also don't feel stressed over cross-mm support: at some point if you > see that separate patch grows we can stop from there, listing all the > cross-mm todos/investigations in the cover letter and start with single-mm. Yes, I think this would be the best way forward (see my previous reply). Thanks, Suren. > > Thanks, > > -- > Peter Xu >
On Mon, Oct 23, 2023 at 10:43:49AM -0700, Suren Baghdasaryan wrote: > > Maybe we should follow what it does with mremap()? Then your current code > > is fine. Maybe that's the better start. > > I think that was the original intention, basically treating remapping > as a write operation. Maybe I should add a comment here to make it > more clear? Please avoid mention "emulate as a write" - this is not a write, e.g., we move a swap entry over without faulting in the page. We also keep the page states, e.g. on hotness. A write will change all of that. Now rethinking with the recently merged WP_ASYNC: we ignore uffd-wp, which means dirty from uffd-wp async tracking POV, that matches with soft-dirty always set. Looks all good. Perhaps something like "Follow mremap() behavior; ignore uffd-wp for now" should work?
On Mon, Oct 23, 2023 at 5:29 AM David Hildenbrand <david@redhat.com> wrote: > > Focusing on validate_remap_areas(): > > > + > > +static int validate_remap_areas(struct vm_area_struct *src_vma, > > + struct vm_area_struct *dst_vma) > > +{ > > + /* Only allow remapping if both have the same access and protection */ > > + if ((src_vma->vm_flags & VM_ACCESS_FLAGS) != (dst_vma->vm_flags & VM_ACCESS_FLAGS) || > > + pgprot_val(src_vma->vm_page_prot) != pgprot_val(dst_vma->vm_page_prot)) > > + return -EINVAL; > > Makes sense. I do wonder about pkey and friends and if we even have to > so anything special. I don't see anything special done for mremap. Do you have something in mind? > > > + > > + /* Only allow remapping if both are mlocked or both aren't */ > > + if ((src_vma->vm_flags & VM_LOCKED) != (dst_vma->vm_flags & VM_LOCKED)) > > + return -EINVAL; > > + > > + if (!(src_vma->vm_flags & VM_WRITE) || !(dst_vma->vm_flags & VM_WRITE)) > > + return -EINVAL; > > Why does one of both need VM_WRITE? If one really needs it, then the > destination (where we're moving stuff to). As you noticed later, both should have VM_WRITE. > > > + > > + /* > > + * Be strict and only allow remap_pages if either the src or > > + * dst range is registered in the userfaultfd to prevent > > + * userland errors going unnoticed. As far as the VM > > + * consistency is concerned, it would be perfectly safe to > > + * remove this check, but there's no useful usage for > > + * remap_pages ouside of userfaultfd registered ranges. This > > + * is after all why it is an ioctl belonging to the > > + * userfaultfd and not a syscall. > > I think the last sentence is the important bit and the comment can > likely be reduced. Ok, I'll look into shortening it. > > > + * > > + * Allow both vmas to be registered in the userfaultfd, just > > + * in case somebody finds a way to make such a case useful. > > + * Normally only one of the two vmas would be registered in > > + * the userfaultfd. > > Should we just check the destination? That makes most sense to me, > because with uffd we are resolving uffd-events. And just like > copy/zeropage we want to resolve a page fault ("userfault") of a > non-present page on the destination. I think that makes sense. Not sure why the original implementation needed the check for source too. Seems unnecessary. > > > > + */ > > + if (!dst_vma->vm_userfaultfd_ctx.ctx && > > + !src_vma->vm_userfaultfd_ctx.ctx) > > + return -EINVAL; > > > > > + > > + /* > > + * FIXME: only allow remapping across anonymous vmas, > > + * tmpfs should be added. > > + */ > > + if (!vma_is_anonymous(src_vma) || !vma_is_anonymous(dst_vma)) > > + return -EINVAL; > > Why a FIXME here? Just drop the comment completely or replace it with > "We only allow to remap anonymous folios accross anonymous VMAs". Will do. I guess Andrea had plans to cover tmpfs as well. > > > + > > + /* > > + * Ensure the dst_vma has a anon_vma or this page > > + * would get a NULL anon_vma when moved in the > > + * dst_vma. > > + */ > > + if (unlikely(anon_vma_prepare(dst_vma))) > > + return -ENOMEM; > > Makes sense. > > > + > > + return 0; > > +} > > Thanks, Suren. > -- > Cheers, > > David / dhildenb > > -- > To unsubscribe from this group and stop receiving emails from it, send an email to kernel-team+unsubscribe@android.com. >
On Mon, Oct 23, 2023 at 8:53 AM David Hildenbrand <david@redhat.com> wrote: > > On 23.10.23 14:29, David Hildenbrand wrote: > >> + > >> + /* Only allow remapping if both are mlocked or both aren't */ > >> + if ((src_vma->vm_flags & VM_LOCKED) != (dst_vma->vm_flags & VM_LOCKED)) > >> + return -EINVAL; > >> + > >> + if (!(src_vma->vm_flags & VM_WRITE) || !(dst_vma->vm_flags & VM_WRITE)) > >> + return -EINVAL; > > > > Why does one of both need VM_WRITE? If one really needs it, then the > > destination (where we're moving stuff to). > > Just realized that we want both to be writable. > > If you have this in place, there is no need to use maybe*_mkwrite(), you > can use the non-maybe variants. Ack. > > I recall that for UFFDIO_COPY we even support PROT_NONE VMAs, is there > any reason why we want to have different semantics here? I don't think so. At least not for the single-mm case. > > -- > Cheers, > > David / dhildenb > > -- > To unsubscribe from this group and stop receiving emails from it, send an email to kernel-team+unsubscribe@android.com. >
On Mon, Oct 23, 2023 at 11:37 AM Peter Xu <peterx@redhat.com> wrote: > > On Mon, Oct 23, 2023 at 10:43:49AM -0700, Suren Baghdasaryan wrote: > > > Maybe we should follow what it does with mremap()? Then your current code > > > is fine. Maybe that's the better start. > > > > I think that was the original intention, basically treating remapping > > as a write operation. Maybe I should add a comment here to make it > > more clear? > > Please avoid mention "emulate as a write" - this is not a write, e.g., we > move a swap entry over without faulting in the page. We also keep the page > states, e.g. on hotness. A write will change all of that. Understood. > > Now rethinking with the recently merged WP_ASYNC: we ignore uffd-wp, which > means dirty from uffd-wp async tracking POV, that matches with soft-dirty > always set. Looks all good. > > Perhaps something like "Follow mremap() behavior; ignore uffd-wp for now" > should work? Sounds good. Will add in the next version. Thanks! > > -- > Peter Xu >
On 23.10.23 20:56, Suren Baghdasaryan wrote: > On Mon, Oct 23, 2023 at 5:29 AM David Hildenbrand <david@redhat.com> wrote: >> >> Focusing on validate_remap_areas(): >> >>> + >>> +static int validate_remap_areas(struct vm_area_struct *src_vma, >>> + struct vm_area_struct *dst_vma) >>> +{ >>> + /* Only allow remapping if both have the same access and protection */ >>> + if ((src_vma->vm_flags & VM_ACCESS_FLAGS) != (dst_vma->vm_flags & VM_ACCESS_FLAGS) || >>> + pgprot_val(src_vma->vm_page_prot) != pgprot_val(dst_vma->vm_page_prot)) >>> + return -EINVAL; >> >> Makes sense. I do wonder about pkey and friends and if we even have to >> so anything special. > > I don't see anything special done for mremap. Do you have something in mind? Nothing concrete, not a pkey expert. But as there is indeed nothing pkey-special in the VMA, there is nothing we can really check for and/or adjust. So let's assume this is fine. >> >>> + >>> + /* Only allow remapping if both are mlocked or both aren't */ >>> + if ((src_vma->vm_flags & VM_LOCKED) != (dst_vma->vm_flags & VM_LOCKED)) >>> + return -EINVAL; >>> + >>> + if (!(src_vma->vm_flags & VM_WRITE) || !(dst_vma->vm_flags & VM_WRITE)) >>> + return -EINVAL; >> >> Why does one of both need VM_WRITE? If one really needs it, then the >> destination (where we're moving stuff to). > > As you noticed later, both should have VM_WRITE. Can you comment why? Just a simplification for now? Would be good to add that comment in the code as well. /* For now, we keep it simple and only move between writable VMAs. */ >>> + */ >>> + if (!dst_vma->vm_userfaultfd_ctx.ctx && >>> + !src_vma->vm_userfaultfd_ctx.ctx) >>> + return -EINVAL; >> >> >> >>> + >>> + /* >>> + * FIXME: only allow remapping across anonymous vmas, >>> + * tmpfs should be added. >>> + */ >>> + if (!vma_is_anonymous(src_vma) || !vma_is_anonymous(dst_vma)) >>> + return -EINVAL; >> >> Why a FIXME here? Just drop the comment completely or replace it with >> "We only allow to remap anonymous folios accross anonymous VMAs". > > Will do. I guess Andrea had plans to cover tmpfs as well. That is rather future work (or what's to fix here?) and better documented in the cover letter. Having thought about VMA checks, I do wonder if we want to just block some VM_ flags right at the beginning (VM_IO,VM_PFNMAP,VM_HUGETLB,...). That might be covered by some other checks here implicitly, but I'm not 100% sure if that's always the case. An explicit list as in vma_ksm_compatible() might be clearer. Further, I wonder if we have to block VM_SHADOW_STACK; we certainly don't want to let users modify the shadow stack by moving modified target pages into place. But this might already be covered by earlier checks (vm_page_prot? but I didn't look up with which setting we ended up in the upstream version). Cc'ing Rick: see "validate_remap_areas()" in [1] [1] https://lkml.kernel.org/r/20231009064230.2952396-3-surenb@google.com
On Tue, Oct 24, 2023 at 7:27 AM David Hildenbrand <david@redhat.com> wrote: > > On 23.10.23 20:56, Suren Baghdasaryan wrote: > > On Mon, Oct 23, 2023 at 5:29 AM David Hildenbrand <david@redhat.com> wrote: > >> > >> Focusing on validate_remap_areas(): > >> > >>> + > >>> +static int validate_remap_areas(struct vm_area_struct *src_vma, > >>> + struct vm_area_struct *dst_vma) > >>> +{ > >>> + /* Only allow remapping if both have the same access and protection */ > >>> + if ((src_vma->vm_flags & VM_ACCESS_FLAGS) != (dst_vma->vm_flags & VM_ACCESS_FLAGS) || > >>> + pgprot_val(src_vma->vm_page_prot) != pgprot_val(dst_vma->vm_page_prot)) > >>> + return -EINVAL; > >> > >> Makes sense. I do wonder about pkey and friends and if we even have to > >> so anything special. > > > > I don't see anything special done for mremap. Do you have something in mind? > > Nothing concrete, not a pkey expert. But as there is indeed nothing > pkey-special in the VMA, there is nothing we can really check for and/or > adjust. > > So let's assume this is fine. Sounds good until someone tells us otherwise. > > >> > >>> + > >>> + /* Only allow remapping if both are mlocked or both aren't */ > >>> + if ((src_vma->vm_flags & VM_LOCKED) != (dst_vma->vm_flags & VM_LOCKED)) > >>> + return -EINVAL; > >>> + > >>> + if (!(src_vma->vm_flags & VM_WRITE) || !(dst_vma->vm_flags & VM_WRITE)) > >>> + return -EINVAL; > >> > >> Why does one of both need VM_WRITE? If one really needs it, then the > >> destination (where we're moving stuff to). > > > > As you noticed later, both should have VM_WRITE. > > Can you comment why? Just a simplification for now? Would be good to add > that comment in the code as well. Yeah, I thought to move a page both areas should be writable since we are technically modifying both by this operation. > > /* For now, we keep it simple and only move between writable VMAs. */ Ack. Will add. > > >>> + */ > >>> + if (!dst_vma->vm_userfaultfd_ctx.ctx && > >>> + !src_vma->vm_userfaultfd_ctx.ctx) > >>> + return -EINVAL; > >> > >> > >> > >>> + > >>> + /* > >>> + * FIXME: only allow remapping across anonymous vmas, > >>> + * tmpfs should be added. > >>> + */ > >>> + if (!vma_is_anonymous(src_vma) || !vma_is_anonymous(dst_vma)) > >>> + return -EINVAL; > >> > >> Why a FIXME here? Just drop the comment completely or replace it with > >> "We only allow to remap anonymous folios accross anonymous VMAs". > > > > Will do. I guess Andrea had plans to cover tmpfs as well. > > > That is rather future work (or what's to fix here?) and better > documented in the cover letter. Ack. > > Having thought about VMA checks, I do wonder if we want to just block > some VM_ flags right at the beginning (VM_IO,VM_PFNMAP,VM_HUGETLB,...). > That might be covered by some other checks here implicitly, but I'm not > 100% sure if that's always the case. An explicit list as in > vma_ksm_compatible() might be clearer. > > Further, I wonder if we have to block VM_SHADOW_STACK; we certainly > don't want to let users modify the shadow stack by moving modified > target pages into place. But this might already be covered by earlier > checks (vm_page_prot? but I didn't look up with which setting we ended > up in the upstream version). Good point. I'll check if existing checks already cover these and if not will add them. Thanks, Suren. > > Cc'ing Rick: see "validate_remap_areas()" in [1] > > [1] https://lkml.kernel.org/r/20231009064230.2952396-3-surenb@google.com > > > -- > Cheers, > > David / dhildenb >
diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst index 203e26da5f92..e5cc8848dcb3 100644 --- a/Documentation/admin-guide/mm/userfaultfd.rst +++ b/Documentation/admin-guide/mm/userfaultfd.rst @@ -113,6 +113,9 @@ events, except page fault notifications, may be generated: areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the analogous feature indicating support for shmem virtual memory areas. +- ``UFFD_FEATURE_MOVE`` indicates that the kernel supports moving an + existing page contents from userspace. + The userland application should set the feature flags it intends to use when invoking the ``UFFDIO_API`` ioctl, to request that those features be enabled if supported. diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index a7c6ef764e63..ac52e0f99a69 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -2039,6 +2039,66 @@ static inline unsigned int uffd_ctx_features(__u64 user_features) return (unsigned int)user_features | UFFD_FEATURE_INITIALIZED; } +static int userfaultfd_remap(struct userfaultfd_ctx *ctx, + unsigned long arg) +{ + __s64 ret; + struct uffdio_move uffdio_move; + struct uffdio_move __user *user_uffdio_move; + struct userfaultfd_wake_range range; + + user_uffdio_move = (struct uffdio_move __user *) arg; + + ret = -EAGAIN; + if (atomic_read(&ctx->mmap_changing)) + goto out; + + ret = -EFAULT; + if (copy_from_user(&uffdio_move, user_uffdio_move, + /* don't copy "remap" last field */ + sizeof(uffdio_move)-sizeof(__s64))) + goto out; + + ret = validate_range(ctx->mm, uffdio_move.dst, uffdio_move.len); + if (ret) + goto out; + + ret = validate_range(current->mm, uffdio_move.src, uffdio_move.len); + if (ret) + goto out; + + ret = -EINVAL; + if (uffdio_move.mode & ~(UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES| + UFFDIO_MOVE_MODE_DONTWAKE)) + goto out; + + if (mmget_not_zero(ctx->mm)) { + ret = remap_pages(ctx->mm, current->mm, + uffdio_move.dst, uffdio_move.src, + uffdio_move.len, uffdio_move.mode); + mmput(ctx->mm); + } else { + return -ESRCH; + } + + if (unlikely(put_user(ret, &user_uffdio_move->move))) + return -EFAULT; + if (ret < 0) + goto out; + + /* len == 0 would wake all */ + BUG_ON(!ret); + range.len = ret; + if (!(uffdio_move.mode & UFFDIO_MOVE_MODE_DONTWAKE)) { + range.start = uffdio_move.dst; + wake_userfault(ctx, &range); + } + ret = range.len == uffdio_move.len ? 0 : -EAGAIN; + +out: + return ret; +} + /* * userland asks for a certain API version and we return which bits * and ioctl commands are implemented in this kernel for such API @@ -2131,6 +2191,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd, case UFFDIO_ZEROPAGE: ret = userfaultfd_zeropage(ctx, arg); break; + case UFFDIO_MOVE: + ret = userfaultfd_remap(ctx, arg); + break; case UFFDIO_WRITEPROTECT: ret = userfaultfd_writeprotect(ctx, arg); break; diff --git a/include/linux/rmap.h b/include/linux/rmap.h index b26fe858fd44..8034eda972e5 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -121,6 +121,11 @@ static inline void anon_vma_lock_write(struct anon_vma *anon_vma) down_write(&anon_vma->root->rwsem); } +static inline int anon_vma_trylock_write(struct anon_vma *anon_vma) +{ + return down_write_trylock(&anon_vma->root->rwsem); +} + static inline void anon_vma_unlock_write(struct anon_vma *anon_vma) { up_write(&anon_vma->root->rwsem); diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index f2dc19f40d05..ce8d20b57e8c 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -93,6 +93,18 @@ extern int mwriteprotect_range(struct mm_struct *dst_mm, extern long uffd_wp_range(struct vm_area_struct *vma, unsigned long start, unsigned long len, bool enable_wp); +/* remap_pages */ +void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2); +void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2); +ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm, + unsigned long dst_start, unsigned long src_start, + unsigned long len, __u64 flags); +int remap_pages_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, + pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, + struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, + unsigned long dst_addr, unsigned long src_addr); + /* mm helpers */ static inline bool is_mergeable_vm_userfaultfd_ctx(struct vm_area_struct *vma, struct vm_userfaultfd_ctx vm_ctx) diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h index 0dbc81015018..2841e4ea8f2c 100644 --- a/include/uapi/linux/userfaultfd.h +++ b/include/uapi/linux/userfaultfd.h @@ -41,7 +41,8 @@ UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \ UFFD_FEATURE_WP_UNPOPULATED | \ UFFD_FEATURE_POISON | \ - UFFD_FEATURE_WP_ASYNC) + UFFD_FEATURE_WP_ASYNC | \ + UFFD_FEATURE_MOVE) #define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \ @@ -50,6 +51,7 @@ ((__u64)1 << _UFFDIO_WAKE | \ (__u64)1 << _UFFDIO_COPY | \ (__u64)1 << _UFFDIO_ZEROPAGE | \ + (__u64)1 << _UFFDIO_MOVE | \ (__u64)1 << _UFFDIO_WRITEPROTECT | \ (__u64)1 << _UFFDIO_CONTINUE | \ (__u64)1 << _UFFDIO_POISON) @@ -73,6 +75,7 @@ #define _UFFDIO_WAKE (0x02) #define _UFFDIO_COPY (0x03) #define _UFFDIO_ZEROPAGE (0x04) +#define _UFFDIO_MOVE (0x05) #define _UFFDIO_WRITEPROTECT (0x06) #define _UFFDIO_CONTINUE (0x07) #define _UFFDIO_POISON (0x08) @@ -92,6 +95,8 @@ struct uffdio_copy) #define UFFDIO_ZEROPAGE _IOWR(UFFDIO, _UFFDIO_ZEROPAGE, \ struct uffdio_zeropage) +#define UFFDIO_MOVE _IOWR(UFFDIO, _UFFDIO_MOVE, \ + struct uffdio_move) #define UFFDIO_WRITEPROTECT _IOWR(UFFDIO, _UFFDIO_WRITEPROTECT, \ struct uffdio_writeprotect) #define UFFDIO_CONTINUE _IOWR(UFFDIO, _UFFDIO_CONTINUE, \ @@ -222,6 +227,9 @@ struct uffdio_api { * asynchronous mode is supported in which the write fault is * automatically resolved and write-protection is un-set. * It implies UFFD_FEATURE_WP_UNPOPULATED. + * + * UFFD_FEATURE_MOVE indicates that the kernel supports moving an + * existing page contents from userspace. */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) #define UFFD_FEATURE_EVENT_FORK (1<<1) @@ -239,6 +247,7 @@ struct uffdio_api { #define UFFD_FEATURE_WP_UNPOPULATED (1<<13) #define UFFD_FEATURE_POISON (1<<14) #define UFFD_FEATURE_WP_ASYNC (1<<15) +#define UFFD_FEATURE_MOVE (1<<16) __u64 features; __u64 ioctls; @@ -347,6 +356,24 @@ struct uffdio_poison { __s64 updated; }; +struct uffdio_move { + __u64 dst; + __u64 src; + __u64 len; + /* + * Especially if used to atomically remove memory from the + * address space the wake on the dst range is not needed. + */ +#define UFFDIO_MOVE_MODE_DONTWAKE ((__u64)1<<0) +#define UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES ((__u64)1<<1) + __u64 mode; + /* + * "move" is written by the ioctl and must be at the end: the + * copy_from_user will not read the last 8 bytes. + */ + __s64 move; +}; + /* * Flags for the userfaultfd(2) system call itself. */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9656be95a542..6fac5c3d66e6 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2086,6 +2086,144 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, return ret; } +#ifdef CONFIG_USERFAULTFD +/* + * The PT lock for src_pmd and the mmap_lock for reading are held by + * the caller, but it must return after releasing the + * page_table_lock. Just move the page from src_pmd to dst_pmd if possible. + * Return zero if succeeded in moving the page, -EAGAIN if it needs to be + * repeated by the caller, or other errors in case of failure. + */ +int remap_pages_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, + pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval, + struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, + unsigned long dst_addr, unsigned long src_addr) +{ + pmd_t _dst_pmd, src_pmdval; + struct page *src_page; + struct folio *src_folio; + struct anon_vma *src_anon_vma; + spinlock_t *src_ptl, *dst_ptl; + pgtable_t src_pgtable, dst_pgtable; + struct mmu_notifier_range range; + int err = 0; + + src_pmdval = *src_pmd; + src_ptl = pmd_lockptr(src_mm, src_pmd); + + lockdep_assert_held(src_ptl); + mmap_assert_locked(src_mm); + mmap_assert_locked(dst_mm); + + BUG_ON(!pmd_none(dst_pmdval)); + BUG_ON(src_addr & ~HPAGE_PMD_MASK); + BUG_ON(dst_addr & ~HPAGE_PMD_MASK); + + if (!pmd_trans_huge(src_pmdval)) { + spin_unlock(src_ptl); + if (is_pmd_migration_entry(src_pmdval)) { + pmd_migration_entry_wait(src_mm, &src_pmdval); + return -EAGAIN; + } + return -ENOENT; + } + + src_page = pmd_page(src_pmdval); + if (unlikely(!PageAnonExclusive(src_page))) { + spin_unlock(src_ptl); + return -EBUSY; + } + + src_folio = page_folio(src_page); + folio_get(src_folio); + spin_unlock(src_ptl); + + /* preallocate dst_pgtable if needed */ + if (dst_mm != src_mm) { + dst_pgtable = pte_alloc_one(dst_mm); + if (unlikely(!dst_pgtable)) { + err = -ENOMEM; + goto put_folio; + } + } else { + dst_pgtable = NULL; + } + + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, src_addr, + src_addr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); + + folio_lock(src_folio); + + /* + * split_huge_page walks the anon_vma chain without the page + * lock. Serialize against it with the anon_vma lock, the page + * lock is not enough. + */ + src_anon_vma = folio_get_anon_vma(src_folio); + if (!src_anon_vma) { + err = -EAGAIN; + goto unlock_folio; + } + anon_vma_lock_write(src_anon_vma); + + dst_ptl = pmd_lockptr(dst_mm, dst_pmd); + double_pt_lock(src_ptl, dst_ptl); + if (unlikely(!pmd_same(*src_pmd, src_pmdval) || + !pmd_same(*dst_pmd, dst_pmdval))) { + double_pt_unlock(src_ptl, dst_ptl); + err = -EAGAIN; + goto put_anon_vma; + } + if (!PageAnonExclusive(&src_folio->page)) { + double_pt_unlock(src_ptl, dst_ptl); + err = -EBUSY; + goto put_anon_vma; + } + + BUG_ON(!folio_test_head(src_folio)); + BUG_ON(!folio_test_anon(src_folio)); + + folio_move_anon_rmap(src_folio, dst_vma); + WRITE_ONCE(src_folio->index, linear_page_index(dst_vma, dst_addr)); + + src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd); + _dst_pmd = mk_huge_pmd(&src_folio->page, dst_vma->vm_page_prot); + _dst_pmd = maybe_pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma); + set_pmd_at(dst_mm, dst_addr, dst_pmd, _dst_pmd); + + src_pgtable = pgtable_trans_huge_withdraw(src_mm, src_pmd); + if (dst_pgtable) { + pgtable_trans_huge_deposit(dst_mm, dst_pmd, dst_pgtable); + pte_free(src_mm, src_pgtable); + dst_pgtable = NULL; + + mm_inc_nr_ptes(dst_mm); + mm_dec_nr_ptes(src_mm); + add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); + add_mm_counter(src_mm, MM_ANONPAGES, -HPAGE_PMD_NR); + } else { + pgtable_trans_huge_deposit(dst_mm, dst_pmd, src_pgtable); + } + double_pt_unlock(src_ptl, dst_ptl); + +put_anon_vma: + anon_vma_unlock_write(src_anon_vma); + put_anon_vma(src_anon_vma); +unlock_folio: + /* unblock rmap walks */ + folio_unlock(src_folio); + mmu_notifier_invalidate_range_end(&range); + if (dst_pgtable) + pte_free(dst_mm, dst_pgtable); +put_folio: + folio_put(src_folio); + + return err; +} +#endif /* CONFIG_USERFAULTFD */ + /* * Returns page table lock pointer if a given pmd maps a thp, NULL otherwise. * diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2b5c0321d96b..0c1ee7172852 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1136,6 +1136,9 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, * Prevent all access to pagetables with the exception of * gup_fast later handled by the ptep_clear_flush and the VM * handled by the anon_vma lock + PG_lock. + * + * UFFDIO_MOVE is prevented to race as well thanks to the + * mmap_lock. */ mmap_write_lock(mm); result = hugepage_vma_revalidate(mm, address, true, &vma, cc); diff --git a/mm/rmap.c b/mm/rmap.c index f9ddc50269d2..a5919cac9a08 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -490,6 +490,12 @@ void __init anon_vma_init(void) * page_remove_rmap() that the anon_vma pointer from page->mapping is valid * if there is a mapcount, we can dereference the anon_vma after observing * those. + * + * NOTE: the caller should normally hold folio lock when calling this. If + * not, the caller needs to double check the anon_vma didn't change after + * taking the anon_vma lock for either read or write (UFFDIO_MOVE can modify it + * concurrently without folio lock protection). See folio_lock_anon_vma_read() + * which has already covered that, and comment above remap_pages(). */ struct anon_vma *folio_get_anon_vma(struct folio *folio) { diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 96d9eae5c7cc..45ce1a8b8ab9 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -842,3 +842,605 @@ int mwriteprotect_range(struct mm_struct *dst_mm, unsigned long start, mmap_read_unlock(dst_mm); return err; } + + +void double_pt_lock(spinlock_t *ptl1, + spinlock_t *ptl2) + __acquires(ptl1) + __acquires(ptl2) +{ + spinlock_t *ptl_tmp; + + if (ptl1 > ptl2) { + /* exchange ptl1 and ptl2 */ + ptl_tmp = ptl1; + ptl1 = ptl2; + ptl2 = ptl_tmp; + } + /* lock in virtual address order to avoid lock inversion */ + spin_lock(ptl1); + if (ptl1 != ptl2) + spin_lock_nested(ptl2, SINGLE_DEPTH_NESTING); + else + __acquire(ptl2); +} + +void double_pt_unlock(spinlock_t *ptl1, + spinlock_t *ptl2) + __releases(ptl1) + __releases(ptl2) +{ + spin_unlock(ptl1); + if (ptl1 != ptl2) + spin_unlock(ptl2); + else + __release(ptl2); +} + + +static int remap_present_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, + struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, + unsigned long dst_addr, unsigned long src_addr, + pte_t *dst_pte, pte_t *src_pte, + pte_t orig_dst_pte, pte_t orig_src_pte, + spinlock_t *dst_ptl, spinlock_t *src_ptl, + struct folio *src_folio) +{ + double_pt_lock(dst_ptl, src_ptl); + + if (!pte_same(*src_pte, orig_src_pte) || + !pte_same(*dst_pte, orig_dst_pte)) { + double_pt_unlock(dst_ptl, src_ptl); + return -EAGAIN; + } + if (folio_test_large(src_folio) || + !PageAnonExclusive(&src_folio->page)) { + double_pt_unlock(dst_ptl, src_ptl); + return -EBUSY; + } + + BUG_ON(!folio_test_anon(src_folio)); + + folio_move_anon_rmap(src_folio, dst_vma); + WRITE_ONCE(src_folio->index, linear_page_index(dst_vma, dst_addr)); + + orig_src_pte = ptep_clear_flush(src_vma, src_addr, src_pte); + orig_dst_pte = mk_pte(&src_folio->page, dst_vma->vm_page_prot); + orig_dst_pte = maybe_mkwrite(pte_mkdirty(orig_dst_pte), dst_vma); + + set_pte_at(dst_mm, dst_addr, dst_pte, orig_dst_pte); + + if (dst_mm != src_mm) { + inc_mm_counter(dst_mm, MM_ANONPAGES); + dec_mm_counter(src_mm, MM_ANONPAGES); + } + + double_pt_unlock(dst_ptl, src_ptl); + + return 0; +} + +static int remap_swap_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, + unsigned long dst_addr, unsigned long src_addr, + pte_t *dst_pte, pte_t *src_pte, + pte_t orig_dst_pte, pte_t orig_src_pte, + spinlock_t *dst_ptl, spinlock_t *src_ptl) +{ + if (!pte_swp_exclusive(orig_src_pte)) + return -EBUSY; + + double_pt_lock(dst_ptl, src_ptl); + + if (!pte_same(*src_pte, orig_src_pte) || + !pte_same(*dst_pte, orig_dst_pte)) { + double_pt_unlock(dst_ptl, src_ptl); + return -EAGAIN; + } + + orig_src_pte = ptep_get_and_clear(src_mm, src_addr, src_pte); + set_pte_at(dst_mm, dst_addr, dst_pte, orig_src_pte); + + if (dst_mm != src_mm) { + inc_mm_counter(dst_mm, MM_SWAPENTS); + dec_mm_counter(src_mm, MM_SWAPENTS); + } + + double_pt_unlock(dst_ptl, src_ptl); + + return 0; +} + +/* + * The mmap_lock for reading is held by the caller. Just move the page + * from src_pmd to dst_pmd if possible, and return true if succeeded + * in moving the page. + */ +static int remap_pages_pte(struct mm_struct *dst_mm, + struct mm_struct *src_mm, + pmd_t *dst_pmd, + pmd_t *src_pmd, + struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, + unsigned long dst_addr, + unsigned long src_addr, + __u64 mode) +{ + swp_entry_t entry; + pte_t orig_src_pte, orig_dst_pte; + pte_t src_folio_pte; + spinlock_t *src_ptl, *dst_ptl; + pte_t *src_pte = NULL; + pte_t *dst_pte = NULL; + + struct folio *src_folio = NULL; + struct anon_vma *src_anon_vma = NULL; + struct mmu_notifier_range range; + int err = 0; + + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, src_mm, + src_addr, src_addr + PAGE_SIZE); + mmu_notifier_invalidate_range_start(&range); +retry: + dst_pte = pte_offset_map_nolock(dst_mm, dst_pmd, dst_addr, &dst_ptl); + + /* Retry if a huge pmd materialized from under us */ + if (unlikely(!dst_pte)) { + err = -EAGAIN; + goto out; + } + + src_pte = pte_offset_map_nolock(src_mm, src_pmd, src_addr, &src_ptl); + + /* + * We held the mmap_lock for reading so MADV_DONTNEED + * can zap transparent huge pages under us, or the + * transparent huge page fault can establish new + * transparent huge pages under us. + */ + if (unlikely(!src_pte)) { + err = -EAGAIN; + goto out; + } + + BUG_ON(pmd_none(*dst_pmd)); + BUG_ON(pmd_none(*src_pmd)); + BUG_ON(pmd_trans_huge(*dst_pmd)); + BUG_ON(pmd_trans_huge(*src_pmd)); + + spin_lock(dst_ptl); + orig_dst_pte = *dst_pte; + spin_unlock(dst_ptl); + if (!pte_none(orig_dst_pte)) { + err = -EEXIST; + goto out; + } + + spin_lock(src_ptl); + orig_src_pte = *src_pte; + spin_unlock(src_ptl); + if (pte_none(orig_src_pte)) { + if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) + err = -ENOENT; + else /* nothing to do to remap a hole */ + err = 0; + goto out; + } + + /* If PTE changed after we locked the folio them start over */ + if (src_folio && unlikely(!pte_same(src_folio_pte, orig_src_pte))) { + err = -EAGAIN; + goto out; + } + + if (pte_present(orig_src_pte)) { + /* + * Pin and lock both source folio and anon_vma. Since we are in + * RCU read section, we can't block, so on contention have to + * unmap the ptes, obtain the lock and retry. + */ + if (!src_folio) { + struct folio *folio; + + /* + * Pin the page while holding the lock to be sure the + * page isn't freed under us + */ + spin_lock(src_ptl); + if (!pte_same(orig_src_pte, *src_pte)) { + spin_unlock(src_ptl); + err = -EAGAIN; + goto out; + } + + folio = vm_normal_folio(src_vma, src_addr, orig_src_pte); + if (!folio || folio_test_large(folio) || + !PageAnonExclusive(&folio->page)) { + spin_unlock(src_ptl); + err = -EBUSY; + goto out; + } + + folio_get(folio); + src_folio = folio; + src_folio_pte = orig_src_pte; + spin_unlock(src_ptl); + + if (!folio_trylock(src_folio)) { + pte_unmap(&orig_src_pte); + pte_unmap(&orig_dst_pte); + src_pte = dst_pte = NULL; + /* now we can block and wait */ + folio_lock(src_folio); + goto retry; + } + } + + if (!src_anon_vma) { + /* + * folio_referenced walks the anon_vma chain + * without the folio lock. Serialize against it with + * the anon_vma lock, the folio lock is not enough. + */ + src_anon_vma = folio_get_anon_vma(src_folio); + if (!src_anon_vma) { + /* page was unmapped from under us */ + err = -EAGAIN; + goto out; + } + if (!anon_vma_trylock_write(src_anon_vma)) { + pte_unmap(&orig_src_pte); + pte_unmap(&orig_dst_pte); + src_pte = dst_pte = NULL; + /* now we can block and wait */ + anon_vma_lock_write(src_anon_vma); + goto retry; + } + } + + err = remap_present_pte(dst_mm, src_mm, dst_vma, src_vma, + dst_addr, src_addr, dst_pte, src_pte, + orig_dst_pte, orig_src_pte, + dst_ptl, src_ptl, src_folio); + } else { + entry = pte_to_swp_entry(orig_src_pte); + if (non_swap_entry(entry)) { + if (is_migration_entry(entry)) { + pte_unmap(&orig_src_pte); + pte_unmap(&orig_dst_pte); + src_pte = dst_pte = NULL; + migration_entry_wait(src_mm, src_pmd, + src_addr); + err = -EAGAIN; + } else + err = -EFAULT; + goto out; + } + + err = remap_swap_pte(dst_mm, src_mm, dst_addr, src_addr, + dst_pte, src_pte, + orig_dst_pte, orig_src_pte, + dst_ptl, src_ptl); + } + +out: + if (src_anon_vma) { + anon_vma_unlock_write(src_anon_vma); + put_anon_vma(src_anon_vma); + } + if (src_folio) { + folio_unlock(src_folio); + folio_put(src_folio); + } + if (dst_pte) + pte_unmap(dst_pte); + if (src_pte) + pte_unmap(src_pte); + mmu_notifier_invalidate_range_end(&range); + + return err; +} + +static int validate_remap_areas(struct vm_area_struct *src_vma, + struct vm_area_struct *dst_vma) +{ + /* Only allow remapping if both have the same access and protection */ + if ((src_vma->vm_flags & VM_ACCESS_FLAGS) != (dst_vma->vm_flags & VM_ACCESS_FLAGS) || + pgprot_val(src_vma->vm_page_prot) != pgprot_val(dst_vma->vm_page_prot)) + return -EINVAL; + + /* Only allow remapping if both are mlocked or both aren't */ + if ((src_vma->vm_flags & VM_LOCKED) != (dst_vma->vm_flags & VM_LOCKED)) + return -EINVAL; + + if (!(src_vma->vm_flags & VM_WRITE) || !(dst_vma->vm_flags & VM_WRITE)) + return -EINVAL; + + /* + * Be strict and only allow remap_pages if either the src or + * dst range is registered in the userfaultfd to prevent + * userland errors going unnoticed. As far as the VM + * consistency is concerned, it would be perfectly safe to + * remove this check, but there's no useful usage for + * remap_pages ouside of userfaultfd registered ranges. This + * is after all why it is an ioctl belonging to the + * userfaultfd and not a syscall. + * + * Allow both vmas to be registered in the userfaultfd, just + * in case somebody finds a way to make such a case useful. + * Normally only one of the two vmas would be registered in + * the userfaultfd. + */ + if (!dst_vma->vm_userfaultfd_ctx.ctx && + !src_vma->vm_userfaultfd_ctx.ctx) + return -EINVAL; + + /* + * FIXME: only allow remapping across anonymous vmas, + * tmpfs should be added. + */ + if (!vma_is_anonymous(src_vma) || !vma_is_anonymous(dst_vma)) + return -EINVAL; + + /* + * Ensure the dst_vma has a anon_vma or this page + * would get a NULL anon_vma when moved in the + * dst_vma. + */ + if (unlikely(anon_vma_prepare(dst_vma))) + return -ENOMEM; + + return 0; +} + +/** + * remap_pages - remap arbitrary anonymous pages of an existing vma + * @dst_start: start of the destination virtual memory range + * @src_start: start of the source virtual memory range + * @len: length of the virtual memory range + * + * remap_pages() remaps arbitrary anonymous pages atomically in zero + * copy. It only works on non shared anonymous pages because those can + * be relocated without generating non linear anon_vmas in the rmap + * code. + * + * It provides a zero copy mechanism to handle userspace page faults. + * The source vma pages should have mapcount == 1, which can be + * enforced by using madvise(MADV_DONTFORK) on src vma. + * + * The thread receiving the page during the userland page fault + * will receive the faulting page in the source vma through the network, + * storage or any other I/O device (MADV_DONTFORK in the source vma + * avoids remap_pages() to fail with -EBUSY if the process forks before + * remap_pages() is called), then it will call remap_pages() to map the + * page in the faulting address in the destination vma. + * + * This userfaultfd command works purely via pagetables, so it's the + * most efficient way to move physical non shared anonymous pages + * across different virtual addresses. Unlike mremap()/mmap()/munmap() + * it does not create any new vmas. The mapping in the destination + * address is atomic. + * + * It only works if the vma protection bits are identical from the + * source and destination vma. + * + * It can remap non shared anonymous pages within the same vma too. + * + * If the source virtual memory range has any unmapped holes, or if + * the destination virtual memory range is not a whole unmapped hole, + * remap_pages() will fail respectively with -ENOENT or -EEXIST. This + * provides a very strict behavior to avoid any chance of memory + * corruption going unnoticed if there are userland race conditions. + * Only one thread should resolve the userland page fault at any given + * time for any given faulting address. This means that if two threads + * try to both call remap_pages() on the same destination address at the + * same time, the second thread will get an explicit error from this + * command. + * + * The command retval will return "len" is successful. The command + * however can be interrupted by fatal signals or errors. If + * interrupted it will return the number of bytes successfully + * remapped before the interruption if any, or the negative error if + * none. It will never return zero. Either it will return an error or + * an amount of bytes successfully moved. If the retval reports a + * "short" remap, the remap_pages() command should be repeated by + * userland with src+retval, dst+reval, len-retval if it wants to know + * about the error that interrupted it. + * + * The UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES flag can be specified to + * prevent -ENOENT errors to materialize if there are holes in the + * source virtual range that is being remapped. The holes will be + * accounted as successfully remapped in the retval of the + * command. This is mostly useful to remap hugepage naturally aligned + * virtual regions without knowing if there are transparent hugepage + * in the regions or not, but preventing the risk of having to split + * the hugepmd during the remap. + * + * If there's any rmap walk that is taking the anon_vma locks without + * first obtaining the folio lock (the only current instance is + * folio_referenced), they will have to verify if the folio->mapping + * has changed after taking the anon_vma lock. If it changed they + * should release the lock and retry obtaining a new anon_vma, because + * it means the anon_vma was changed by remap_pages() before the lock + * could be obtained. This is the only additional complexity added to + * the rmap code to provide this anonymous page remapping functionality. + */ +ssize_t remap_pages(struct mm_struct *dst_mm, struct mm_struct *src_mm, + unsigned long dst_start, unsigned long src_start, + unsigned long len, __u64 mode) +{ + struct vm_area_struct *src_vma, *dst_vma; + unsigned long src_addr, dst_addr; + pmd_t *src_pmd, *dst_pmd; + long err = -EINVAL; + ssize_t moved = 0; + + /* + * Sanitize the command parameters: + */ + BUG_ON(src_start & ~PAGE_MASK); + BUG_ON(dst_start & ~PAGE_MASK); + BUG_ON(len & ~PAGE_MASK); + + /* Does the address range wrap, or is the span zero-sized? */ + BUG_ON(src_start + len <= src_start); + BUG_ON(dst_start + len <= dst_start); + + /* + * Because these are read sempahores there's no risk of lock + * inversion. + */ + mmap_read_lock(dst_mm); + if (dst_mm != src_mm) + mmap_read_lock(src_mm); + + /* + * Make sure the vma is not shared, that the src and dst remap + * ranges are both valid and fully within a single existing + * vma. + */ + src_vma = find_vma(src_mm, src_start); + if (!src_vma || (src_vma->vm_flags & VM_SHARED)) + goto out; + if (src_start < src_vma->vm_start || + src_start + len > src_vma->vm_end) + goto out; + + dst_vma = find_vma(dst_mm, dst_start); + if (!dst_vma || (dst_vma->vm_flags & VM_SHARED)) + goto out; + if (dst_start < dst_vma->vm_start || + dst_start + len > dst_vma->vm_end) + goto out; + + err = validate_remap_areas(src_vma, dst_vma); + if (err) + goto out; + + for (src_addr = src_start, dst_addr = dst_start; + src_addr < src_start + len;) { + spinlock_t *ptl; + pmd_t dst_pmdval; + unsigned long step_size; + + BUG_ON(dst_addr >= dst_start + len); + /* + * Below works because anonymous area would not have a + * transparent huge PUD. If file-backed support is added, + * that case would need to be handled here. + */ + src_pmd = mm_find_pmd(src_mm, src_addr); + if (unlikely(!src_pmd)) { + if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) { + err = -ENOENT; + break; + } + src_pmd = mm_alloc_pmd(src_mm, src_addr); + if (unlikely(!src_pmd)) { + err = -ENOMEM; + break; + } + } + dst_pmd = mm_alloc_pmd(dst_mm, dst_addr); + if (unlikely(!dst_pmd)) { + err = -ENOMEM; + break; + } + + dst_pmdval = pmdp_get_lockless(dst_pmd); + /* + * If the dst_pmd is mapped as THP don't override it and just + * be strict. If dst_pmd changes into TPH after this check, the + * remap_pages_huge_pmd() will detect the change and retry + * while remap_pages_pte() will detect the change and fail. + */ + if (unlikely(pmd_trans_huge(dst_pmdval))) { + err = -EEXIST; + break; + } + + ptl = pmd_trans_huge_lock(src_pmd, src_vma); + if (ptl) { + if (pmd_devmap(*src_pmd)) { + spin_unlock(ptl); + err = -ENOENT; + break; + } + + /* + * Check if we can move the pmd without + * splitting it. First check the address + * alignment to be the same in src/dst. These + * checks don't actually need the PT lock but + * it's good to do it here to optimize this + * block away at build time if + * CONFIG_TRANSPARENT_HUGEPAGE is not set. + */ + if ((src_addr & ~HPAGE_PMD_MASK) || (dst_addr & ~HPAGE_PMD_MASK) || + src_start + len - src_addr < HPAGE_PMD_SIZE || !pmd_none(dst_pmdval)) { + spin_unlock(ptl); + split_huge_pmd(src_vma, src_pmd, src_addr); + continue; + } + + err = remap_pages_huge_pmd(dst_mm, src_mm, + dst_pmd, src_pmd, + dst_pmdval, + dst_vma, src_vma, + dst_addr, src_addr); + step_size = HPAGE_PMD_SIZE; + } else { + if (pmd_none(*src_pmd)) { + if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES)) { + err = -ENOENT; + break; + } + if (unlikely(__pte_alloc(src_mm, src_pmd))) { + err = -ENOMEM; + break; + } + } + + if (unlikely(pte_alloc(dst_mm, dst_pmd))) { + err = -ENOMEM; + break; + } + + err = remap_pages_pte(dst_mm, src_mm, + dst_pmd, src_pmd, + dst_vma, src_vma, + dst_addr, src_addr, + mode); + step_size = PAGE_SIZE; + } + + cond_resched(); + + if (fatal_signal_pending(current)) { + /* Do not override an error */ + if (!err || err == -EAGAIN) + err = -EINTR; + break; + } + + if (err) { + if (err == -EAGAIN) + continue; + break; + } + + /* Proceed to the next page */ + dst_addr += step_size; + src_addr += step_size; + moved += step_size; + } + +out: + mmap_read_unlock(dst_mm); + if (dst_mm != src_mm) + mmap_read_unlock(src_mm); + BUG_ON(moved < 0); + BUG_ON(err > 0); + BUG_ON(!moved && !err); + return moved ? moved : err; +}