| Message ID | 20200224203057.162467-1-walken@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Fine grained MM locking | expand |
> Hi, > > This is the first version of my work towards fine grained MM locking. > This is still early work - I am happy with my page fault changes, > but want to expand on the mmap/munmap side of things before I send the > next version. I have previously shared this with some of the copied folks > (for those who received that, there are no additional changes in this > public resend). Please expect a v2 within a few weeks, with further > changes for fine grained range locking in the mmap and munmap paths. hello, Michel. I noticed rwsem has been renamed to mmap_lock and some apis were created for taking the lock. but is the original fine grained mm locking series still under development? maybe i missed something but i failed to find v2 for it. Thanks Barry
On Mon, Mar 21, 2022 at 11:08:48AM +1300, Barry Song wrote: > hello, Michel. I noticed rwsem has been renamed to mmap_lock and > some apis were created for taking the lock. > but is the original fine grained mm locking series still under > development? maybe i missed something but i failed to find v2 > for it. Most recently posted as 20220128131006.67712-1-michel@lespinasse.org
On Mon, Mar 21, 2022 at 12:14 PM Matthew Wilcox <willy@infradead.org> wrote: > > On Mon, Mar 21, 2022 at 11:08:48AM +1300, Barry Song wrote: > > hello, Michel. I noticed rwsem has been renamed to mmap_lock and > > some apis were created for taking the lock. > > but is the original fine grained mm locking series still under > > development? maybe i missed something but i failed to find v2 > > for it. > > Most recently posted as 20220128131006.67712-1-michel@lespinasse.org thanks!
Hi, This is the first version of my work towards fine grained MM locking. This is still early work - I am happy with my page fault changes, but want to expand on the mmap/munmap side of things before I send the next version. I have previously shared this with some of the copied folks (for those who received that, there are no additional changes in this public resend). Please expect a v2 within a few weeks, with further changes for fine grained range locking in the mmap and munmap paths. This work originated in discussions at LSF/MM 2019; it is intended to address the latency issues that are caused by false conflicts between threads working on separate parts of their address space. The priorities are to keep things as simple as possible, and to allow for progressive conversion of the code base to finer grained MM locks. The general approach is to replace the mmap_sem rwsem with a range lock. Initially all lock/unlock sites are automatically converted to lock the entire address space through a new API. Then, the API is extended to support range locking. Locking sites can then be progressively converted to use range locking, while leaving unconverted sites working with no code changes. When using a range lock (as opposed to a coarse lock), the following rules apply: - Some structures (notably the vma rbtree and associated statistics) are per-mm. They need to be locked separately using a new mm_vma_lock. The entire point of this patch set is to reduce false sharing latencies, so the mm_vma_lock must be held only for short times. We expect to do O(log N) operations holding the lock (for example, walking or updating the vma rbtree) but no O(N) operations (such as iterating on all vmas within a range or all mapped pages within a range). - Code holding the mm_vma_lock should only update vma attributes for the range it has a write lock for. However, range locks only protects the vma's attributes, not the vmas themselves - vmas can still be split or merged with their neighbors if they have compatible attributes. - Code holding a range lock but not the mm_vma_lock must be prepared for the vmas at both ends of the locked range to be merged with their neighbors outside of the locked range. The easiest way to do that is to copy the vma of record into a pseudo-vma before releasing the mm_vma_lock (this is a bit kludgy and I would prefer to copy only the necessary VMA attributes, but using a pseudo-vma makes it easier to maintain this patchset out of mainline for the moment). Call sites that take a range lock usualy immediately take the mm_vma_lock next - it would probably be more efficient to collapse mm_vma_lock with the mutex that protects the range lock structures. This isn't done yet as I tried to simplify the initial implementation. In the future I would also like to remove the various workarounds we have been doing to limit mmap_sem hold times (i.e. FAULT_FLAG_ALLOW_RETRY, vm_populate and munmap downgrading to a read lock, ...) which shouldn't be necessary if the locking was only effective on the memory ranges affected by each operation. The included changes apply on top of upstream kernel v5.5. Please apply with git am -p0 - I'm not sure why my git format-patch setup requires that. Commits 1 to 6 implement a range locking API: - 1 implements coarse locking as wrappers around rwsem; - 2 converts most mmap_sem locking sites to use the new coarse locking API (using coccinelle to automate the conversion); - 3 converts remaining mmap_sem locking sites which were missed by coccinelle; - 4 extends the API to support range locking. The initial implementation still uses coarse locking (ignoring the range); but it validates that the callers use matching ranges in lock and unlock calls; - 5 prepares callers to allow for sleeping during unlock; - 6 actually implements the range locking functions. Commits 7 to 12 allow the x86 fault handler to specify a range that may be released while handling the fault: - 7 adds a range field to struct mm_fault; - 8 makes handle_mm_fault() populate that field; - 9 and 10 honor it when dropping mmap_sem during fault handling; - 11 is a cleanup to the x86 fault handler to prepare for 12; - 12 changes the x86 fault handler to use an explicit lock range. Commits 13 to 15 prepare for operating on a pseudo-vma during faults: - 13 adds a prepare_vma_fault which may update the vma of record (specifically, allocate an anon_vma) before creating the pseudo-vma; - 14 disables swap vma readahead as its implementation keeps stats in the vma; - 15 changes the x86 fault handler to use pseudo-vmas when handling anon vmas. Commits 16 and 17 implement range locking in x86 anonymous vma faults: - Commit 16 adds the vma locking API to be used to manipulate vmas when holding a fine grained ranged lock; - Commit 17 converts the x86 fault handler to use a pmd sized range lock when operating on anon vmas. Commits 18 to 20 extend the above to also work on filemap based files: - Commit 18 makes sure we release the correct range when dropping mmap_sem during filemap file access; - Commit 19 tags vm_operations that support range locking; - Commit 20 makes the x86 fault handler use fine grained ranges when faulting the supported files. Commits 21 to 24 implement range locking for the most basic mmap() case: - 21 adds a locked argument to do_mmap(); - 22 makes do_mmap acquire the mmap_sem if locked is false; - 23 converts soem easy call sites to pass locked=false; - 24 changes do_mmap to acquire a fine grained lock in the easiest case (anonymous mapping, known address, no prior existing mapping). Michel Lespinasse (24): MM locking API: initial implementation as rwsem wrappers MM locking API: use coccinelle to convert mmap_sem rwsem call sites MM locking API: manual conversion of mmap_sem call sites missed by coccinelle MM locking API: add range arguments MM locking API: allow for sleeping during unlock MM locking API: implement fine grained range locks mm/memory: add range field to struct vm_fault mm/memory: allow specifying MM lock range to handle_mm_fault() do_swap_page: use the vmf->range field when dropping mmap_sem handle_userfault: use the vmf->range field when dropping mmap_sem x86 fault handler: merge bad_area() functions x86 fault handler: use an explicit MM lock range mm/memory: add prepare_mm_fault() function mm/swap_state: disable swap vma readahead x86 fault handler: use a pseudo-vma when operating on anonymous vmas. MM locking API: add vma locking API x86 fault handler: implement range locking shared file mappings: use the vmf->range field when dropping mmap_sem mm: add field to annotate vm_operations that support range locking x86 fault handler: extend range locking to supported file vmas do_mmap: add locked argument do_mmap: implement locked argument do_mmap: use locked=false in vm_mmap_pgoff() and aio_setup_ring() do_mmap: implement easiest cases of fine grained locking arch/alpha/kernel/traps.c | 4 +- arch/alpha/mm/fault.c | 10 +- arch/arc/kernel/process.c | 4 +- arch/arc/kernel/troubleshoot.c | 4 +- arch/arc/mm/fault.c | 4 +- arch/arm/kernel/process.c | 4 +- arch/arm/kernel/swp_emulate.c | 4 +- arch/arm/lib/uaccess_with_memcpy.c | 16 +- arch/arm/mm/fault.c | 6 +- arch/arm64/kernel/traps.c | 4 +- arch/arm64/kernel/vdso.c | 8 +- arch/arm64/mm/fault.c | 8 +- arch/csky/kernel/vdso.c | 4 +- arch/csky/mm/fault.c | 8 +- arch/hexagon/kernel/vdso.c | 4 +- arch/hexagon/mm/vm_fault.c | 8 +- arch/ia64/kernel/perfmon.c | 8 +- arch/ia64/mm/fault.c | 8 +- arch/ia64/mm/init.c | 12 +- arch/m68k/kernel/sys_m68k.c | 14 +- arch/m68k/mm/fault.c | 8 +- arch/microblaze/mm/fault.c | 12 +- arch/mips/kernel/traps.c | 4 +- arch/mips/kernel/vdso.c | 4 +- arch/mips/mm/fault.c | 10 +- arch/nds32/kernel/vdso.c | 6 +- arch/nds32/mm/fault.c | 12 +- arch/nios2/mm/fault.c | 12 +- arch/nios2/mm/init.c | 4 +- arch/openrisc/mm/fault.c | 10 +- arch/parisc/kernel/traps.c | 6 +- arch/parisc/mm/fault.c | 8 +- arch/powerpc/kernel/vdso.c | 6 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 6 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 12 +- arch/powerpc/kvm/e500_mmu_host.c | 4 +- arch/powerpc/mm/book3s64/iommu_api.c | 4 +- arch/powerpc/mm/book3s64/subpage_prot.c | 12 +- arch/powerpc/mm/copro_fault.c | 4 +- arch/powerpc/mm/fault.c | 12 +- arch/powerpc/oprofile/cell/spu_task_sync.c | 6 +- arch/powerpc/platforms/cell/spufs/file.c | 4 +- arch/riscv/kernel/vdso.c | 4 +- arch/riscv/mm/fault.c | 10 +- arch/s390/kernel/vdso.c | 4 +- arch/s390/kvm/gaccess.c | 4 +- arch/s390/kvm/kvm-s390.c | 24 +- arch/s390/kvm/priv.c | 32 +- arch/s390/mm/fault.c | 6 +- arch/s390/mm/gmap.c | 40 +- arch/s390/pci/pci_mmio.c | 4 +- arch/sh/kernel/sys_sh.c | 6 +- arch/sh/kernel/vsyscall/vsyscall.c | 4 +- arch/sh/mm/fault.c | 14 +- arch/sparc/mm/fault_32.c | 18 +- arch/sparc/mm/fault_64.c | 12 +- arch/sparc/vdso/vma.c | 4 +- arch/um/include/asm/mmu_context.h | 6 +- arch/um/kernel/tlb.c | 2 +- arch/um/kernel/trap.c | 6 +- arch/unicore32/mm/fault.c | 6 +- arch/x86/entry/vdso/vma.c | 10 +- arch/x86/kernel/tboot.c | 2 +- arch/x86/kernel/vm86_32.c | 4 +- arch/x86/kvm/mmu/paging_tmpl.h | 8 +- arch/x86/mm/debug_pagetables.c | 8 +- arch/x86/mm/fault.c | 110 ++- arch/x86/mm/mpx.c | 15 +- arch/x86/um/vdso/vma.c | 4 +- arch/xtensa/mm/fault.c | 10 +- drivers/android/binder_alloc.c | 10 +- drivers/firmware/efi/efi.c | 2 +- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 4 +- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 10 +- drivers/gpu/drm/amd/amdkfd/kfd_events.c | 4 +- drivers/gpu/drm/i915/gem/i915_gem_mman.c | 4 +- drivers/gpu/drm/i915/gem/i915_gem_userptr.c | 8 +- drivers/gpu/drm/nouveau/nouveau_svm.c | 20 +- drivers/gpu/drm/radeon/radeon_cs.c | 4 +- drivers/gpu/drm/radeon/radeon_gem.c | 6 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 4 +- drivers/infiniband/core/umem.c | 6 +- drivers/infiniband/core/umem_odp.c | 10 +- drivers/infiniband/core/uverbs_main.c | 4 +- drivers/infiniband/hw/mlx4/mr.c | 4 +- drivers/infiniband/hw/qib/qib_user_pages.c | 6 +- drivers/infiniband/hw/usnic/usnic_uiom.c | 4 +- drivers/infiniband/sw/siw/siw_mem.c | 4 +- drivers/iommu/amd_iommu_v2.c | 4 +- drivers/iommu/intel-svm.c | 4 +- drivers/media/v4l2-core/videobuf-core.c | 4 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 4 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 4 +- drivers/misc/cxl/cxllib.c | 4 +- drivers/misc/cxl/fault.c | 4 +- drivers/misc/sgi-gru/grufault.c | 16 +- drivers/misc/sgi-gru/grufile.c | 4 +- drivers/oprofile/buffer_sync.c | 10 +- drivers/staging/kpc2000/kpc_dma/fileops.c | 4 +- drivers/tee/optee/call.c | 4 +- drivers/vfio/vfio_iommu_type1.c | 12 +- drivers/xen/gntdev.c | 4 +- drivers/xen/privcmd.c | 14 +- fs/aio.c | 16 +- fs/coredump.c | 4 +- fs/exec.c | 16 +- fs/ext4/file.c | 1 + fs/io_uring.c | 4 +- fs/proc/base.c | 18 +- fs/proc/task_mmu.c | 28 +- fs/proc/task_nommu.c | 18 +- fs/userfaultfd.c | 28 +- include/linux/hugetlb.h | 5 +- include/linux/mm.h | 56 +- include/linux/mm_lock.h | 285 ++++++++ include/linux/mm_types.h | 22 + include/linux/mm_types_task.h | 21 + include/linux/mmu_notifier.h | 5 +- include/linux/pagemap.h | 7 +- include/linux/sched.h | 2 + init/init_task.c | 1 + ipc/shm.c | 11 +- kernel/acct.c | 4 +- kernel/bpf/stackmap.c | 32 +- kernel/events/core.c | 4 +- kernel/events/uprobes.c | 16 +- kernel/exit.c | 8 +- kernel/fork.c | 17 +- kernel/futex.c | 4 +- kernel/sched/fair.c | 4 +- kernel/sys.c | 18 +- kernel/trace/trace_output.c | 4 +- mm/Kconfig | 25 + mm/Makefile | 2 + mm/filemap.c | 10 +- mm/frame_vector.c | 4 +- mm/gup.c | 20 +- mm/hugetlb.c | 13 +- mm/init-mm.c | 3 +- mm/internal.h | 2 +- mm/khugepaged.c | 37 +- mm/ksm.c | 34 +- mm/madvise.c | 18 +- mm/memcontrol.c | 8 +- mm/memory.c | 55 +- mm/mempolicy.c | 22 +- mm/migrate.c | 8 +- mm/mincore.c | 4 +- mm/mlock.c | 16 +- mm/mm_lock_range.c | 691 ++++++++++++++++++ mm/mm_lock_rwsem_checked.c | 134 ++++ mm/mmap.c | 170 +++-- mm/mmu_notifier.c | 4 +- mm/mprotect.c | 12 +- mm/mremap.c | 6 +- mm/msync.c | 8 +- mm/nommu.c | 36 +- mm/oom_kill.c | 4 +- mm/process_vm_access.c | 4 +- mm/shmem.c | 1 + mm/swap_state.c | 6 + mm/swapfile.c | 4 +- mm/userfaultfd.c | 14 +- mm/util.c | 14 +- net/ipv4/tcp.c | 4 +- net/xdp/xdp_umem.c | 4 +- virt/kvm/arm/mmu.c | 14 +- virt/kvm/async_pf.c | 4 +- virt/kvm/kvm_main.c | 8 +- 170 files changed, 2183 insertions(+), 798 deletions(-) create mode 100644 include/linux/mm_lock.h create mode 100644 mm/mm_lock_range.c create mode 100644 mm/mm_lock_rwsem_checked.c