diff mbox series

[v4,1/2] mm, hugepages: add mremap() support for hugepage backed vma

Message ID 20211006194515.423539-1-almasrymina@google.com (mailing list archive)
State New
Headers show
Series [v4,1/2] mm, hugepages: add mremap() support for hugepage backed vma | expand

Commit Message

Mina Almasry Oct. 6, 2021, 7:45 p.m. UTC 
Support mremap() for hugepage backed vma segment by simply repositioning
page table entries. The page table entries are repositioned to the new
virtual address on mremap().

Hugetlb mremap() support is of course generic; my motivating use case
is a library (hugepage_text), which reloads the ELF text of executables
in hugepages. This significantly increases the execution performance of
said executables.

Restricts the mremap operation on hugepages to up to the size of the
original mapping as the underlying hugetlb reservation is not yet
capable of handling remapping to a larger size.

During the mremap() operation we detect pmd_share'd mappings and we
unshare those during the mremap(). On access and fault the sharing is
established again.

Signed-off-by: Mina Almasry <almasrymina@google.com>

Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: Ken Chen <kenchen@google.com>
Cc: Chris Kennelly <ckennelly@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Vlastimil Babka <vbabka@suse.cz>
Cc: Kirill Shutemov <kirill@shutemov.name>


---

Changes in v4:
- Added addr, new_addr, old_len, and new_len hugepage alignment.

Changes in v3:
- Addressed review comments from Mike.
- Separated tests into their own patch.

Changes in v2:
- Re-wrote comment around clear_vma_resv_huge_pages() to make it clear
that the resv_map has been moved to the new VMA and why we need to clear it
from the current VMA.
- We detect huge_pmd_shared() pte's and unshare those rather than bug on
hugetlb_vma_shareable().
- This case now returns EFAULT:
if (!vma || vma->vm_start > addr)
	 goto out;
- Added kselftests for mremap() support.

---
 include/linux/hugetlb.h |  27 +++++++++
 mm/hugetlb.c            | 131 +++++++++++++++++++++++++++++++++++++---
 mm/mremap.c             |  32 +++++++++-
 3 files changed, 179 insertions(+), 11 deletions(-)

--
2.33.0.882.g93a45727a2-goog

Comments

kernel test robot Oct. 6, 2021, 11:24 p.m. UTC | #1 
Hi Mina,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on hnaz-mm/master]

url:    https://github.com/0day-ci/linux/commits/Mina-Almasry/mm-hugepages-add-mremap-support-for-hugepage-backed-vma/20211007-034701
base:   https://github.com/hnaz/linux-mm master
config: um-x86_64_defconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/3829eb526dc386bfb3870172413086cd8f874738
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Mina-Almasry/mm-hugepages-add-mremap-support-for-hugepage-backed-vma/20211007-034701
        git checkout 3829eb526dc386bfb3870172413086cd8f874738
        # save the attached .config to linux build tree
        mkdir build_dir
        make W=1 O=build_dir ARCH=um SUBARCH=x86_64 SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All error/warnings (new ones prefixed by >>):

   ld: kernel/sysctl.o: in function `hugetlb_vma_shareable':
>> sysctl.c:(.text+0x1dc0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/core.o: in function `hugetlb_vma_shareable':
   core.c:(.text+0x7a0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/loadavg.o: in function `hugetlb_vma_shareable':
   loadavg.c:(.text+0x0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/clock.o: in function `hugetlb_vma_shareable':
   clock.c:(.text+0x20): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/cputime.o: in function `hugetlb_vma_shareable':
   cputime.c:(.text+0x0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/idle.o: in function `hugetlb_vma_shareable':
   idle.c:(.text+0x110): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/fair.o: in function `hugetlb_vma_shareable':
   fair.c:(.text+0x2010): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/rt.o: in function `hugetlb_vma_shareable':
   rt.c:(.text+0x1040): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/deadline.o: in function `hugetlb_vma_shareable':
   deadline.c:(.text+0x2610): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/wait.o: in function `hugetlb_vma_shareable':
   wait.c:(.text+0xa70): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/wait_bit.o: in function `hugetlb_vma_shareable':
   wait_bit.c:(.text+0x240): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/swait.o: in function `hugetlb_vma_shareable':
   swait.c:(.text+0x3f0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/completion.o: in function `hugetlb_vma_shareable':
   completion.c:(.text+0x180): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/debug.o: in function `hugetlb_vma_shareable':
   debug.c:(.text+0x0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/cpuacct.o: in function `hugetlb_vma_shareable':
   cpuacct.c:(.text+0x400): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: kernel/sched/membarrier.o: in function `hugetlb_vma_shareable':
   membarrier.c:(.text+0x0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/filemap.o: in function `hugetlb_vma_shareable':
   filemap.c:(.text+0x47f0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/folio-compat.o: in function `hugetlb_vma_shareable':
   folio-compat.c:(.text+0x2a0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/swap.o: in function `hugetlb_vma_shareable':
   swap.c:(.text+0x1a30): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/vmscan.o: in function `hugetlb_vma_shareable':
   vmscan.c:(.text+0x24c0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/shmem.o: in function `hugetlb_vma_shareable':
   shmem.c:(.text+0x52b0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/util.o: in function `hugetlb_vma_shareable':
   util.c:(.text+0xcb0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/compaction.o: in function `hugetlb_vma_shareable':
   compaction.c:(.text+0x0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/debug.o: in function `hugetlb_vma_shareable':
   debug.c:(.text+0x1b0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/gup.o: in function `hugetlb_vma_shareable':
   gup.c:(.text+0x800): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/memory.o: in function `hugetlb_vma_shareable':
   memory.c:(.text+0xc70): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/mincore.o: in function `hugetlb_vma_shareable':
   mincore.c:(.text+0x2b0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/mlock.o: in function `hugetlb_vma_shareable':
   mlock.c:(.text+0x740): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/mmap.o: in function `hugetlb_vma_shareable':
   mmap.c:(.text+0xa20): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/mprotect.o: in function `hugetlb_vma_shareable':
   mprotect.c:(.text+0x0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/mremap.o: in function `hugetlb_vma_shareable':
   mremap.c:(.text+0x210): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/page_vma_mapped.o: in function `hugetlb_vma_shareable':
   page_vma_mapped.c:(.text+0x0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/pagewalk.o: in function `hugetlb_vma_shareable':
   pagewalk.c:(.text+0x5c0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/pgtable-generic.o: in function `hugetlb_vma_shareable':
   pgtable-generic.c:(.text+0x0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/rmap.o: in function `hugetlb_vma_shareable':
   rmap.c:(.text+0x7d0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/vmalloc.o: in function `hugetlb_vma_shareable':
   vmalloc.c:(.text+0x2870): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/page_alloc.o: in function `hugetlb_vma_shareable':
   page_alloc.c:(.text+0x1bb0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/madvise.o: in function `hugetlb_vma_shareable':
   madvise.c:(.text+0x1770): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/swap_state.o: in function `hugetlb_vma_shareable':
   swap_state.c:(.text+0x120): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/swapfile.o: in function `hugetlb_vma_shareable':
   swapfile.c:(.text+0x1050): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: mm/memfd.o: in function `hugetlb_vma_shareable':
   memfd.c:(.text+0x4a0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: fs/aio.o: in function `hugetlb_vma_shareable':
   aio.c:(.text+0x37f0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: fs/io_uring.o: in function `hugetlb_vma_shareable':
   io_uring.c:(.text+0xdb10): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: fs/binfmt_elf.o: in function `hugetlb_vma_shareable':
   binfmt_elf.c:(.text+0x3190): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: fs/iomap/buffered-io.o: in function `hugetlb_vma_shareable':
   buffered-io.c:(.text+0x3730): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: fs/proc/task_mmu.o: in function `hugetlb_vma_shareable':
   task_mmu.c:(.text+0x1f90): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: fs/proc/array.o: in function `hugetlb_vma_shareable':
   array.c:(.text+0x70): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: fs/proc/meminfo.o: in function `hugetlb_vma_shareable':
   meminfo.c:(.text+0x50): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: fs/proc/page.o: in function `hugetlb_vma_shareable':
   page.c:(.text+0x1c0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: ipc/shm.o: in function `hugetlb_vma_shareable':
   shm.c:(.text+0x14c0): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
   ld: security/commoncap.o: in function `hugetlb_vma_shareable':
   commoncap.c:(.text+0x70): multiple definition of `hugetlb_vma_shareable'; kernel/fork.o:fork.c:(.text+0xc90): first defined here
--
   In file included from kernel/fork.c:51:
>> include/linux/hugetlb.h:417:6: warning: no previous prototype for 'hugetlb_vma_shareable' [-Wmissing-prototypes]
     417 | bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
         |      ^~~~~~~~~~~~~~~~~~~~~
   kernel/fork.c:161:13: warning: no previous prototype for 'arch_release_task_struct' [-Wmissing-prototypes]
     161 | void __weak arch_release_task_struct(struct task_struct *tsk)
         |             ^~~~~~~~~~~~~~~~~~~~~~~~
   kernel/fork.c:814:20: warning: no previous prototype for 'arch_task_cache_init' [-Wmissing-prototypes]
     814 | void __init __weak arch_task_cache_init(void) { }
         |                    ^~~~~~~~~~~~~~~~~~~~
   kernel/fork.c:909:12: warning: no previous prototype for 'arch_dup_task_struct' [-Wmissing-prototypes]
     909 | int __weak arch_dup_task_struct(struct task_struct *dst,
         |            ^~~~~~~~~~~~~~~~~~~~
   In file included from kernel/fork.c:51:
   include/linux/hugetlb.h:240:12: warning: 'huge_pmd_shared' defined but not used [-Wunused-function]
     240 | static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
         |            ^~~~~~~~~~~~~~~
--
   In file included from kernel/sysctl.c:46:
>> include/linux/hugetlb.h:417:6: warning: no previous prototype for 'hugetlb_vma_shareable' [-Wmissing-prototypes]
     417 | bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
         |      ^~~~~~~~~~~~~~~~~~~~~
   In file included from kernel/sysctl.c:46:
   include/linux/hugetlb.h:240:12: warning: 'huge_pmd_shared' defined but not used [-Wunused-function]
     240 | static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
         |            ^~~~~~~~~~~~~~~
--
   In file included from include/linux/migrate.h:8,
                    from kernel/sched/sched.h:53,
                    from kernel/sched/core.c:13:
>> include/linux/hugetlb.h:417:6: warning: no previous prototype for 'hugetlb_vma_shareable' [-Wmissing-prototypes]
     417 | bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
         |      ^~~~~~~~~~~~~~~~~~~~~
   kernel/sched/core.c: In function 'ttwu_stat':
   kernel/sched/core.c:3482:13: warning: variable 'rq' set but not used [-Wunused-but-set-variable]
    3482 |  struct rq *rq;
         |             ^~
   In file included from include/linux/migrate.h:8,
                    from kernel/sched/sched.h:53,
                    from kernel/sched/core.c:13:
   At top level:
   include/linux/hugetlb.h:240:12: warning: 'huge_pmd_shared' defined but not used [-Wunused-function]
     240 | static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
         |            ^~~~~~~~~~~~~~~
--
   In file included from include/linux/migrate.h:8,
                    from kernel/sched/sched.h:53,
                    from kernel/sched/fair.c:23:
>> include/linux/hugetlb.h:417:6: warning: no previous prototype for 'hugetlb_vma_shareable' [-Wmissing-prototypes]
     417 | bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
         |      ^~~~~~~~~~~~~~~~~~~~~
   kernel/sched/fair.c:631:5: warning: no previous prototype for 'sched_update_scaling' [-Wmissing-prototypes]
     631 | int sched_update_scaling(void)
         |     ^~~~~~~~~~~~~~~~~~~~
   kernel/sched/fair.c: In function 'update_curr':
   kernel/sched/fair.c:860:28: warning: variable 'stats' set but not used [-Wunused-but-set-variable]
     860 |   struct sched_statistics *stats;
         |                            ^~~~~
   kernel/sched/fair.c: In function 'update_stats_wait_start_fair':
   kernel/sched/fair.c:893:22: warning: variable 'p' set but not used [-Wunused-but-set-variable]
     893 |  struct task_struct *p = NULL;
         |                      ^
   kernel/sched/fair.c:892:27: warning: variable 'stats' set but not used [-Wunused-but-set-variable]
     892 |  struct sched_statistics *stats;
         |                           ^~~~~
   kernel/sched/fair.c: In function 'update_stats_wait_end_fair':
   kernel/sched/fair.c:910:22: warning: variable 'p' set but not used [-Wunused-but-set-variable]
     910 |  struct task_struct *p = NULL;
         |                      ^
   kernel/sched/fair.c:909:27: warning: variable 'stats' set but not used [-Wunused-but-set-variable]
     909 |  struct sched_statistics *stats;
         |                           ^~~~~
   kernel/sched/fair.c: In function 'update_stats_enqueue_sleeper_fair':
   kernel/sched/fair.c:936:22: warning: variable 'tsk' set but not used [-Wunused-but-set-variable]
     936 |  struct task_struct *tsk = NULL;
         |                      ^~~
   kernel/sched/fair.c:935:27: warning: variable 'stats' set but not used [-Wunused-but-set-variable]
     935 |  struct sched_statistics *stats;
         |                           ^~~~~
   kernel/sched/fair.c: In function 'set_next_entity':
   kernel/sched/fair.c:4451:28: warning: variable 'stats' set but not used [-Wunused-but-set-variable]
    4451 |   struct sched_statistics *stats;
         |                            ^~~~~
   In file included from include/linux/migrate.h:8,
                    from kernel/sched/sched.h:53,
                    from kernel/sched/fair.c:23:
   At top level:
   include/linux/hugetlb.h:240:12: warning: 'huge_pmd_shared' defined but not used [-Wunused-function]
     240 | static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
         |            ^~~~~~~~~~~~~~~
--
   In file included from include/linux/migrate.h:8,
                    from kernel/sched/sched.h:53,
                    from kernel/sched/rt.c:6:
>> include/linux/hugetlb.h:417:6: warning: no previous prototype for 'hugetlb_vma_shareable' [-Wmissing-prototypes]
     417 | bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
         |      ^~~~~~~~~~~~~~~~~~~~~
   kernel/sched/rt.c:669:6: warning: no previous prototype for 'sched_rt_bandwidth_account' [-Wmissing-prototypes]
     669 | bool sched_rt_bandwidth_account(struct rt_rq *rt_rq)
         |      ^~~~~~~~~~~~~~~~~~~~~~~~~~
   kernel/sched/rt.c: In function 'update_stats_wait_start_rt':
   kernel/sched/rt.c:1292:22: warning: variable 'p' set but not used [-Wunused-but-set-variable]
    1292 |  struct task_struct *p = NULL;
         |                      ^
   kernel/sched/rt.c: In function 'update_stats_enqueue_sleeper_rt':
   kernel/sched/rt.c:1311:22: warning: variable 'p' set but not used [-Wunused-but-set-variable]
    1311 |  struct task_struct *p = NULL;
         |                      ^
   kernel/sched/rt.c: In function 'update_stats_wait_end_rt':
   kernel/sched/rt.c:1341:22: warning: variable 'p' set but not used [-Wunused-but-set-variable]
    1341 |  struct task_struct *p = NULL;
         |                      ^
   In file included from include/linux/migrate.h:8,
                    from kernel/sched/sched.h:53,
                    from kernel/sched/rt.c:6:
   At top level:
   include/linux/hugetlb.h:240:12: warning: 'huge_pmd_shared' defined but not used [-Wunused-function]
     240 | static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
         |            ^~~~~~~~~~~~~~~
--
   In file included from include/linux/migrate.h:8,
                    from kernel/sched/sched.h:53,
                    from kernel/sched/deadline.c:18:
>> include/linux/hugetlb.h:417:6: warning: no previous prototype for 'hugetlb_vma_shareable' [-Wmissing-prototypes]
     417 | bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
         |      ^~~~~~~~~~~~~~~~~~~~~
   kernel/sched/deadline.c: In function 'update_stats_wait_start_dl':
   kernel/sched/deadline.c:1486:27: warning: variable 'stats' set but not used [-Wunused-but-set-variable]
    1486 |  struct sched_statistics *stats;
         |                           ^~~~~
   kernel/sched/deadline.c: In function 'update_stats_wait_end_dl':
   kernel/sched/deadline.c:1498:27: warning: variable 'stats' set but not used [-Wunused-but-set-variable]
    1498 |  struct sched_statistics *stats;
         |                           ^~~~~
   kernel/sched/deadline.c: In function 'update_stats_enqueue_sleeper_dl':
   kernel/sched/deadline.c:1510:27: warning: variable 'stats' set but not used [-Wunused-but-set-variable]
    1510 |  struct sched_statistics *stats;
         |                           ^~~~~
   In file included from include/linux/migrate.h:8,
                    from kernel/sched/sched.h:53,
                    from kernel/sched/deadline.c:18:
   At top level:
   include/linux/hugetlb.h:240:12: warning: 'huge_pmd_shared' defined but not used [-Wunused-function]
     240 | static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
         |            ^~~~~~~~~~~~~~~
--
   In file included from include/linux/migrate.h:8,
                    from kernel/sched/sched.h:53,
                    from kernel/sched/debug.c:9:
>> include/linux/hugetlb.h:417:6: warning: no previous prototype for 'hugetlb_vma_shareable' [-Wmissing-prototypes]
     417 | bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
         |      ^~~~~~~~~~~~~~~~~~~~~
   kernel/sched/debug.c: In function 'print_cfs_group_stats':
   kernel/sched/debug.c:466:41: warning: unused variable 'stats' [-Wunused-variable]
     466 |                struct sched_statistics *stats =  __schedstats_from_se(se);
         |                                         ^~~~~
   In file included from include/linux/migrate.h:8,
                    from kernel/sched/sched.h:53,
                    from kernel/sched/debug.c:9:
   At top level:
   include/linux/hugetlb.h:240:12: warning: 'huge_pmd_shared' defined but not used [-Wunused-function]
     240 | static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
         |            ^~~~~~~~~~~~~~~
--
   In file included from mm/vmalloc.c:40:
>> include/linux/hugetlb.h:417:6: warning: no previous prototype for 'hugetlb_vma_shareable' [-Wmissing-prototypes]
     417 | bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
         |      ^~~~~~~~~~~~~~~~~~~~~
   mm/vmalloc.c:1654:6: warning: no previous prototype for 'set_iounmap_nonlazy' [-Wmissing-prototypes]
    1654 | void set_iounmap_nonlazy(void)
         |      ^~~~~~~~~~~~~~~~~~~
   In file included from mm/vmalloc.c:40:
   include/linux/hugetlb.h:240:12: warning: 'huge_pmd_shared' defined but not used [-Wunused-function]
     240 | static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
         |            ^~~~~~~~~~~~~~~
--
   In file included from include/linux/migrate.h:8,
                    from mm/page_alloc.c:61:
>> include/linux/hugetlb.h:417:6: warning: no previous prototype for 'hugetlb_vma_shareable' [-Wmissing-prototypes]
     417 | bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
         |      ^~~~~~~~~~~~~~~~~~~~~
   mm/page_alloc.c:3802:15: warning: no previous prototype for 'should_fail_alloc_page' [-Wmissing-prototypes]
    3802 | noinline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
         |               ^~~~~~~~~~~~~~~~~~~~~~
   In file included from include/linux/migrate.h:8,
                    from mm/page_alloc.c:61:
   include/linux/hugetlb.h:240:12: warning: 'huge_pmd_shared' defined but not used [-Wunused-function]
     240 | static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
         |            ^~~~~~~~~~~~~~~
--
   In file included from fs/io_uring.c:71:
>> include/linux/hugetlb.h:417:6: warning: no previous prototype for 'hugetlb_vma_shareable' [-Wmissing-prototypes]
     417 | bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
         |      ^~~~~~~~~~~~~~~~~~~~~
   fs/io_uring.c: In function 'io_queue_async_work':
   fs/io_uring.c:1491:61: warning: parameter 'locked' set but not used [-Wunused-but-set-parameter]
    1491 | static void io_queue_async_work(struct io_kiocb *req, bool *locked)
         |                                                       ~~~~~~^~~~~~
   fs/io_uring.c: In function '__io_submit_flush_completions':
   fs/io_uring.c:2334:33: warning: variable 'prev' set but not used [-Wunused-but-set-variable]
    2334 |  struct io_wq_work_node *node, *prev;
         |                                 ^~~~
   In file included from fs/io_uring.c:71:
   At top level:
   include/linux/hugetlb.h:240:12: warning: 'huge_pmd_shared' defined but not used [-Wunused-function]
     240 | static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
         |            ^~~~~~~~~~~~~~~
..

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
kernel test robot Oct. 7, 2021, 12:14 a.m. UTC | #2 
Hi Mina,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on hnaz-mm/master]

url:    https://github.com/0day-ci/linux/commits/Mina-Almasry/mm-hugepages-add-mremap-support-for-hugepage-backed-vma/20211007-034701
base:   https://github.com/hnaz/linux-mm master
config: nds32-buildonly-randconfig-r004-20211004 (attached as .config)
compiler: nds32le-linux-gcc (GCC) 11.2.0
reproduce (this is a W=1 build):
        wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
        chmod +x ~/bin/make.cross
        # https://github.com/0day-ci/linux/commit/3829eb526dc386bfb3870172413086cd8f874738
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Mina-Almasry/mm-hugepages-add-mremap-support-for-hugepage-backed-vma/20211007-034701
        git checkout 3829eb526dc386bfb3870172413086cd8f874738
        # save the attached .config to linux build tree
        mkdir build_dir
        COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-11.2.0 make.cross O=build_dir ARCH=nds32 SHELL=/bin/bash

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>

All errors (new ones prefixed by >>):

   nds32le-linux-ld: fs/btrfs/inode.o: in function `hugetlb_vma_shareable':
>> inode.c:(.text+0x4eec): multiple definition of `hugetlb_vma_shareable'; fs/btrfs/disk-io.o:disk-io.c:(.text+0x3c24): first defined here

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all@lists.01.org
Mina Almasry Oct. 7, 2021, 12:18 a.m. UTC | #3 
On Wed, Oct 6, 2021 at 5:15 PM kernel test robot <lkp@intel.com> wrote:
>
> Hi Mina,
>
> Thank you for the patch! Yet something to improve:
>
> [auto build test ERROR on hnaz-mm/master]
>

My apologies. I'll look into this and submit a v5 with a fix.
diff mbox series

Patch

diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index ebaba02706c87..6b500d4fc643b 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -124,6 +124,7 @@  struct hugepage_subpool *hugepage_new_subpool(struct hstate *h, long max_hpages,
 void hugepage_put_subpool(struct hugepage_subpool *spool);

 void reset_vma_resv_huge_pages(struct vm_area_struct *vma);
+void clear_vma_resv_huge_pages(struct vm_area_struct *vma);
 int hugetlb_sysctl_handler(struct ctl_table *, int, void *, size_t *, loff_t *);
 int hugetlb_overcommit_handler(struct ctl_table *, int, void *, size_t *,
 		loff_t *);
@@ -132,6 +133,10 @@  int hugetlb_treat_movable_handler(struct ctl_table *, int, void *, size_t *,
 int hugetlb_mempolicy_sysctl_handler(struct ctl_table *, int, void *, size_t *,
 		loff_t *);

+int move_hugetlb_page_tables(struct vm_area_struct *vma,
+			     struct vm_area_struct *new_vma,
+			     unsigned long old_addr, unsigned long new_addr,
+			     unsigned long len);
 int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_area_struct *);
 long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *,
 			 struct page **, struct vm_area_struct **,
@@ -187,6 +192,7 @@  pte_t *huge_pte_offset(struct mm_struct *mm,
 		       unsigned long addr, unsigned long sz);
 int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
 				unsigned long *addr, pte_t *ptep);
+int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep);
 void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
 				unsigned long *start, unsigned long *end);
 struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address,
@@ -208,6 +214,7 @@  unsigned long hugetlb_change_protection(struct vm_area_struct *vma,

 bool is_hugetlb_entry_migration(pte_t pte);
 void hugetlb_unshare_all_pmds(struct vm_area_struct *vma);
+bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr);

 #else /* !CONFIG_HUGETLB_PAGE */

@@ -215,6 +222,10 @@  static inline void reset_vma_resv_huge_pages(struct vm_area_struct *vma)
 {
 }

+static inline void clear_vma_resv_huge_pages(struct vm_area_struct *vma)
+{
+}
+
 static inline unsigned long hugetlb_total_pages(void)
 {
 	return 0;
@@ -226,6 +237,11 @@  static inline struct address_space *hugetlb_page_mapping_lock_write(
 	return NULL;
 }

+static int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
+{
+	return 0;
+}
+
 static inline int huge_pmd_unshare(struct mm_struct *mm,
 					struct vm_area_struct *vma,
 					unsigned long *addr, pte_t *ptep)
@@ -262,6 +278,12 @@  static inline int copy_hugetlb_page_range(struct mm_struct *dst,
 	return 0;
 }

+#define move_hugetlb_page_tables(vma, new_vma, old_addr, new_addr, len)        \
+	({                                                                     \
+		BUG();                                                         \
+		0;                                                             \
+	})
+
 static inline void hugetlb_report_meminfo(struct seq_file *m)
 {
 }
@@ -392,6 +414,11 @@  static inline vm_fault_t hugetlb_fault(struct mm_struct *mm,

 static inline void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) { }

+bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
+{
+	return 0;
+}
+
 #endif /* !CONFIG_HUGETLB_PAGE */
 /*
  * hugepages at page global directory. If arch support
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 6d2f4c25dd9fb..8200b4c8d09d8 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1015,6 +1015,35 @@  void reset_vma_resv_huge_pages(struct vm_area_struct *vma)
 		vma->vm_private_data = (void *)0;
 }

+/*
+ * Reset and decrement one ref on hugepage private reservation.
+ * Called with mm->mmap_sem writer semaphore held.
+ * This function should be only used by move_vma() and operate on
+ * same sized vma. It should never come here with last ref on the
+ * reservation.
+ */
+void clear_vma_resv_huge_pages(struct vm_area_struct *vma)
+{
+	/*
+	 * Clear the old hugetlb private page reservation.
+	 * It has already been transferred to new_vma.
+	 *
+	 * During a mremap() operation of a hugetlb vma we call move_vma()
+	 * which copies *vma* into *new_vma* and unmaps *vma*. After the copy
+	 * operation both *new_vma* and *vma* share a reference to the resv_map
+	 * struct, and at that point *vma* is about to be unmapped. We don't
+	 * want to return the reservation to the pool at unmap of *vma* because
+	 * the reservation still lives on in new_vma, so simply decrement the
+	 * ref here and remove the resv_map reference from this vma.
+	 */
+	struct resv_map *reservations = vma_resv_map(vma);
+
+	if (reservations && is_vma_resv_set(vma, HPAGE_RESV_OWNER))
+		kref_put(&reservations->refs, resv_map_release);
+
+	reset_vma_resv_huge_pages(vma);
+}
+
 /* Returns true if the VMA has associated reserve pages */
 static bool vma_has_reserves(struct vm_area_struct *vma, long chg)
 {
@@ -4800,6 +4829,82 @@  int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
 	return ret;
 }

+static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
+			  unsigned long new_addr, pte_t *src_pte)
+{
+	struct hstate *h = hstate_vma(vma);
+	struct mm_struct *mm = vma->vm_mm;
+	pte_t *dst_pte, pte;
+	spinlock_t *src_ptl, *dst_ptl;
+
+	dst_pte = huge_pte_offset(mm, new_addr, huge_page_size(h));
+	dst_ptl = huge_pte_lock(h, mm, dst_pte);
+	src_ptl = huge_pte_lockptr(h, mm, src_pte);
+
+	/*
+	 * We don't have to worry about the ordering of src and dst ptlocks
+	 * because exclusive mmap_sem (or the i_mmap_lock) prevents deadlock.
+	 */
+	if (src_ptl != dst_ptl)
+		spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
+
+	pte = huge_ptep_get_and_clear(mm, old_addr, src_pte);
+	set_huge_pte_at(mm, new_addr, dst_pte, pte);
+
+	if (src_ptl != dst_ptl)
+		spin_unlock(src_ptl);
+	spin_unlock(dst_ptl);
+}
+
+int move_hugetlb_page_tables(struct vm_area_struct *vma,
+			     struct vm_area_struct *new_vma,
+			     unsigned long old_addr, unsigned long new_addr,
+			     unsigned long len)
+{
+	struct hstate *h = hstate_vma(vma);
+	struct address_space *mapping = vma->vm_file->f_mapping;
+	unsigned long sz = huge_page_size(h);
+	struct mm_struct *mm = vma->vm_mm;
+	unsigned long old_end = old_addr + len;
+	unsigned long old_addr_copy;
+	pte_t *src_pte, *dst_pte;
+	struct mmu_notifier_range range;
+
+	mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, vma, mm, old_addr,
+				old_end);
+	adjust_range_if_pmd_sharing_possible(vma, &range.start, &range.end);
+	mmu_notifier_invalidate_range_start(&range);
+	/* Prevent race with file truncation */
+	i_mmap_lock_write(mapping);
+	for (; old_addr < old_end; old_addr += sz, new_addr += sz) {
+		src_pte = huge_pte_offset(mm, old_addr, sz);
+		if (!src_pte)
+			continue;
+		if (huge_pte_none(huge_ptep_get(src_pte)))
+			continue;
+
+		/* old_addr arg to huge_pmd_unshare() is a pointer and so the
+		 * arg may be modified. Pass a copy instead to preserve the
+		 * value in old_arg.
+		 */
+		old_addr_copy = old_addr;
+
+		if (huge_pmd_unshare(mm, vma, &old_addr_copy, src_pte))
+			continue;
+
+		dst_pte = huge_pte_alloc(mm, new_vma, new_addr, sz);
+		if (!dst_pte)
+			break;
+
+		move_huge_pte(vma, old_addr, new_addr, src_pte);
+	}
+	i_mmap_unlock_write(mapping);
+	flush_tlb_range(vma, old_end - len, old_end);
+	mmu_notifier_invalidate_range_end(&range);
+
+	return len + old_addr - old_end;
+}
+
 static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
 				   unsigned long start, unsigned long end,
 				   struct page *ref_page)
@@ -6280,7 +6385,7 @@  static unsigned long page_table_shareable(struct vm_area_struct *svma,
 	return saddr;
 }

-static bool vma_shareable(struct vm_area_struct *vma, unsigned long addr)
+bool hugetlb_vma_shareable(struct vm_area_struct *vma, unsigned long addr)
 {
 	unsigned long base = addr & PUD_MASK;
 	unsigned long end = base + PUD_SIZE;
@@ -6299,7 +6404,7 @@  bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr)
 	if (uffd_disable_huge_pmd_share(vma))
 		return false;
 #endif
-	return vma_shareable(vma, addr);
+	return hugetlb_vma_shareable(vma, addr);
 }

 /*
@@ -6339,12 +6444,6 @@  void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
  * sharing is possible.  For hugetlbfs, this prevents removal of any page
  * table entries associated with the address space.  This is important as we
  * are setting up sharing based on existing page table entries (mappings).
- *
- * NOTE: This routine is only called from huge_pte_alloc.  Some callers of
- * huge_pte_alloc know that sharing is not possible and do not take
- * i_mmap_rwsem as a performance optimization.  This is handled by the
- * if !vma_shareable check at the beginning of the routine. i_mmap_rwsem is
- * only required for subsequent processing.
  */
 pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
 		      unsigned long addr, pud_t *pud)
@@ -6422,7 +6521,23 @@  int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
 	return 1;
 }

+int huge_pmd_shared(struct vm_area_struct *vma, pte_t *ptep)
+{
+	i_mmap_assert_write_locked(vma->vm_file->f_mapping);
+	BUG_ON(page_count(virt_to_page(ptep)) == 0);
+	if (page_count(virt_to_page(ptep)) == 1)
+		return 0;
+
+	return 1;
+}
+
 #else /* !CONFIG_ARCH_WANT_HUGE_PMD_SHARE */
+static bool hugetlb_vma_shareable(struct vm_area_struct *vma,
+				  unsigned long addr)
+{
+	return false;
+}
+
 pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma,
 		      unsigned long addr, pud_t *pud)
 {
diff --git a/mm/mremap.c b/mm/mremap.c
index c0b6c41b7b78f..6a3f7d38b7539 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -489,6 +489,10 @@  unsigned long move_page_tables(struct vm_area_struct *vma,
 	old_end = old_addr + len;
 	flush_cache_range(vma, old_addr, old_end);

+	if (is_vm_hugetlb_page(vma))
+		return move_hugetlb_page_tables(vma, new_vma, old_addr,
+						new_addr, len);
+
 	mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, vma, vma->vm_mm,
 				old_addr, old_end);
 	mmu_notifier_invalidate_range_start(&range);
@@ -646,6 +650,10 @@  static unsigned long move_vma(struct vm_area_struct *vma,
 		mremap_userfaultfd_prep(new_vma, uf);
 	}

+	if (is_vm_hugetlb_page(vma)) {
+		clear_vma_resv_huge_pages(vma);
+	}
+
 	/* Conceal VM_ACCOUNT so old reservation is not undone */
 	if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP)) {
 		vma->vm_flags &= ~VM_ACCOUNT;
@@ -739,9 +747,6 @@  static struct vm_area_struct *vma_to_resize(unsigned long addr,
 			(vma->vm_flags & (VM_DONTEXPAND | VM_PFNMAP)))
 		return ERR_PTR(-EINVAL);

-	if (is_vm_hugetlb_page(vma))
-		return ERR_PTR(-EINVAL);
-
 	/* We can't remap across vm area boundaries */
 	if (old_len > vma->vm_end - addr)
 		return ERR_PTR(-EFAULT);
@@ -937,6 +942,27 @@  SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len,

 	if (mmap_write_lock_killable(current->mm))
 		return -EINTR;
+	vma = find_vma(mm, addr);
+	if (!vma || vma->vm_start > addr) {
+		ret = EFAULT;
+		goto out;
+	}
+
+	if (is_vm_hugetlb_page(vma)) {
+		struct hstate *h __maybe_unused = hstate_vma(vma);
+
+		old_len = ALIGN(old_len, huge_page_size(h));
+		new_len = ALIGN(new_len, huge_page_size(h));
+		addr = ALIGN(addr, huge_page_size(h));
+		new_addr = ALIGN(new_addr, huge_page_size(h));
+
+		/*
+		 * Don't allow remap expansion, because the underlying hugetlb
+		 * reservation is not yet capable to handle split reservation.
+		 */
+		if (new_len > old_len)
+			goto out;
+	}

 	if (flags & (MREMAP_FIXED | MREMAP_DONTUNMAP)) {
 		ret = mremap_to(addr, old_len, new_addr, new_len,