[v8,0/4] introduce memory hinting API for external process
mbox series

Message ID 20200622192900.22757-1-minchan@kernel.org
Headers show
Series
  • introduce memory hinting API for external process
Related show

Message

Minchan Kim June 22, 2020, 7:28 p.m. UTC
Now, we have MADV_PAGEOUT and MADV_COLD as madvise hinting API. With that,
application could give hints to kernel what memory range are preferred to be
reclaimed. However, in some platform(e.g., Android), the information
required to make the hinting decision is not known to the app.
Instead, it is known to a centralized userspace daemon(e.g., ActivityManagerService),
and that daemon must be able to initiate reclaim on its own without any app
involvement.

To solve the concern, this patch introduces new syscall - process_madvise(2).
Bascially, it's same with madvise(2) syscall but it has some differences.

1. It needs pidfd of target process to provide the hint
2. It supports only MADV_{COLD|PAGEOUT} at this moment.
   Other hints in madvise will be opened when there are explicit requests from
   community to prevent unexpected bugs we couldn't support.
3. Only privileged processes can do something for other process's address
   space.

For more detail of the new API, please see "mm: introduce external memory hinting API"
description in this patchset.

* from v7 -  http://lore.kernel.org/r/20200302193630.68771-1-minchan@kernel.org
  * dropping pid support from new syscall and fold releated patches into syscall patch
  * dropping KSM patch by discussion - Oleksandr, I lost the discussion.
    Please resend the single patch against of the patchset if you resolves the discussion.
    https://lore.kernel.org/linux-api/20200302193630.68771-8-minchan@kernel.org/

* from v6 - https://lore.kernel.org/linux-api/20200219014433.88424-1-minchan@kernel.org/
  * fix comments and descriptions - Suren
  * Add Reviewed-by - Suren
  * fix build break reported by 0-day

* from v5 - https://lore.kernel.org/linux-mm/20200214170520.160271-1-minchan@kernel.org/
  * use null task and requestor's mm for io_madvise - Jann and Jens
  * use right commit description for moving pidfd_get_pid - Christoph

* from v4 - https://lore.kernel.org/linux-mm/20200212233946.246210-1-minchan@kernel.org/
  * pass mm down to functions, not accessing task->mm - Jann
  * clean up - Alexander
  * add Reviewed-by - Alexander, SeongJae
  * patch reordering

* from v3 - https://lore.kernel.org/linux-mm/20200128001641.5086-1-minchan@kernel.org/
  * verify task->mm aftere access_mm - Oleg
  * split some patches for easy review - Alexander
  * clean up fatal signal checking - Suren

* from v2 - https://lore.kernel.org/linux-mm/20200116235953.163318-1-minchan@kernel.org/
  * check signal callee and caller to bail out - Kirill Tkhai
  * put more clarification for justification of new API

* from v1 - https://lore.kernel.org/linux-mm/20200110213433.94739-1-minchan@kernel.org/
  * fix syscall number - SeongJae
  * use get_pid_task - Kirill Tkhai
  * extend API to support pid as well as pidfd - Kirill Tkhai

Minchan Kim (4):
  mm/madvise: pass task and mm to do_madvise
  pid: move pidfd_get_pid() to pid.c
  mm/madvise: introduce process_madvise() syscall: an external memory
    hinting API
  mm/madvise: check fatal signal pending of target process

 arch/alpha/kernel/syscalls/syscall.tbl      |   1 +
 arch/arm/tools/syscall.tbl                  |   1 +
 arch/arm64/include/asm/unistd.h             |   2 +-
 arch/arm64/include/asm/unistd32.h           |   2 +
 arch/ia64/kernel/syscalls/syscall.tbl       |   1 +
 arch/m68k/kernel/syscalls/syscall.tbl       |   1 +
 arch/microblaze/kernel/syscalls/syscall.tbl |   1 +
 arch/mips/kernel/syscalls/syscall_n32.tbl   |   1 +
 arch/mips/kernel/syscalls/syscall_n64.tbl   |   1 +
 arch/mips/kernel/syscalls/syscall_o32.tbl   |   1 +
 arch/parisc/kernel/syscalls/syscall.tbl     |   1 +
 arch/powerpc/kernel/syscalls/syscall.tbl    |   1 +
 arch/s390/kernel/syscalls/syscall.tbl       |   1 +
 arch/sh/kernel/syscalls/syscall.tbl         |   1 +
 arch/sparc/kernel/syscalls/syscall.tbl      |   1 +
 arch/x86/entry/syscalls/syscall_32.tbl      |   1 +
 arch/x86/entry/syscalls/syscall_64.tbl      |   2 +
 arch/xtensa/kernel/syscalls/syscall.tbl     |   1 +
 fs/io_uring.c                               |   2 +-
 include/linux/compat.h                      |   4 +
 include/linux/mm.h                          |   3 +-
 include/linux/pid.h                         |   1 +
 include/linux/syscalls.h                    |   2 +
 include/uapi/asm-generic/unistd.h           |   4 +-
 kernel/exit.c                               |  17 --
 kernel/pid.c                                |  17 ++
 kernel/sys_ni.c                             |   2 +
 mm/madvise.c                                | 190 +++++++++++++++++---
 28 files changed, 217 insertions(+), 46 deletions(-)

Comments

Minchan Kim June 22, 2020, 7:36 p.m. UTC | #1
On Mon, Jun 22, 2020 at 12:28:56PM -0700, Minchan Kim wrote:
> Now, we have MADV_PAGEOUT and MADV_COLD as madvise hinting API. With that,
> application could give hints to kernel what memory range are preferred to be
> reclaimed. However, in some platform(e.g., Android), the information
> required to make the hinting decision is not known to the app.
> Instead, it is known to a centralized userspace daemon(e.g., ActivityManagerService),
> and that daemon must be able to initiate reclaim on its own without any app
> involvement.
> 
> To solve the concern, this patch introduces new syscall - process_madvise(2).
> Bascially, it's same with madvise(2) syscall but it has some differences.
> 
> 1. It needs pidfd of target process to provide the hint
> 2. It supports only MADV_{COLD|PAGEOUT} at this moment.
>    Other hints in madvise will be opened when there are explicit requests from
>    community to prevent unexpected bugs we couldn't support.
> 3. Only privileged processes can do something for other process's address
>    space.
> 
> For more detail of the new API, please see "mm: introduce external memory hinting API"
> description in this patchset.
> 
> * from v7 -  http://lore.kernel.org/r/20200302193630.68771-1-minchan@kernel.org
>   * dropping pid support from new syscall and fold releated patches into syscall patch
>   * dropping KSM patch by discussion - Oleksandr, I lost the discussion.
>     Please resend the single patch against of the patchset if you resolves the discussion.
>     https://lore.kernel.org/linux-api/20200302193630.68771-8-minchan@kernel.org/

Oleksandr, it seems you discussed something with Vlastimil but couldn't
find conclustion yet and Since Jann put an a new note in the thread,
I detach the patch from this patchset.

Please send the KSM patch based on this patchset if you belive there is
no need to be actionable for comments.
Oleksandr Natalenko June 23, 2020, 8:59 a.m. UTC | #2
Hello.

On Mon, Jun 22, 2020 at 12:36:35PM -0700, Minchan Kim wrote:
> On Mon, Jun 22, 2020 at 12:28:56PM -0700, Minchan Kim wrote:
> > Now, we have MADV_PAGEOUT and MADV_COLD as madvise hinting API. With that,
> > application could give hints to kernel what memory range are preferred to be
> > reclaimed. However, in some platform(e.g., Android), the information
> > required to make the hinting decision is not known to the app.
> > Instead, it is known to a centralized userspace daemon(e.g., ActivityManagerService),
> > and that daemon must be able to initiate reclaim on its own without any app
> > involvement.
> > 
> > To solve the concern, this patch introduces new syscall - process_madvise(2).
> > Bascially, it's same with madvise(2) syscall but it has some differences.
> > 
> > 1. It needs pidfd of target process to provide the hint
> > 2. It supports only MADV_{COLD|PAGEOUT} at this moment.
> >    Other hints in madvise will be opened when there are explicit requests from
> >    community to prevent unexpected bugs we couldn't support.
> > 3. Only privileged processes can do something for other process's address
> >    space.
> > 
> > For more detail of the new API, please see "mm: introduce external memory hinting API"
> > description in this patchset.
> > 
> > * from v7 -  http://lore.kernel.org/r/20200302193630.68771-1-minchan@kernel.org
> >   * dropping pid support from new syscall and fold releated patches into syscall patch
> >   * dropping KSM patch by discussion - Oleksandr, I lost the discussion.
> >     Please resend the single patch against of the patchset if you resolves the discussion.
> >     https://lore.kernel.org/linux-api/20200302193630.68771-8-minchan@kernel.org/
> 
> Oleksandr, it seems you discussed something with Vlastimil but couldn't
> find conclustion yet and Since Jann put an a new note in the thread,
> I detach the patch from this patchset.
> 
> Please send the KSM patch based on this patchset if you belive there is
> no need to be actionable for comments.

I don't think we came to some definite conclusion, but I'll try to implement
Vlastimil's suggestion and then send it separately for evaluation.

Lets keep KSM bit out of your submission so I won't slow it down.
Oleksandr Natalenko June 23, 2020, 9:07 a.m. UTC | #3
On Mon, Jun 22, 2020 at 12:28:56PM -0700, Minchan Kim wrote:
> Now, we have MADV_PAGEOUT and MADV_COLD as madvise hinting API. With that,
> application could give hints to kernel what memory range are preferred to be
> reclaimed. However, in some platform(e.g., Android), the information
> required to make the hinting decision is not known to the app.
> Instead, it is known to a centralized userspace daemon(e.g., ActivityManagerService),
> and that daemon must be able to initiate reclaim on its own without any app
> involvement.
> 
> To solve the concern, this patch introduces new syscall - process_madvise(2).
> Bascially, it's same with madvise(2) syscall but it has some differences.
> 
> 1. It needs pidfd of target process to provide the hint
> 2. It supports only MADV_{COLD|PAGEOUT} at this moment.
>    Other hints in madvise will be opened when there are explicit requests from
>    community to prevent unexpected bugs we couldn't support.
> 3. Only privileged processes can do something for other process's address
>    space.
> 
> For more detail of the new API, please see "mm: introduce external memory hinting API"
> description in this patchset.
> 
> * from v7 -  http://lore.kernel.org/r/20200302193630.68771-1-minchan@kernel.org
>   * dropping pid support from new syscall and fold releated patches into syscall patch
>   * dropping KSM patch by discussion - Oleksandr, I lost the discussion.
>     Please resend the single patch against of the patchset if you resolves the discussion.
>     https://lore.kernel.org/linux-api/20200302193630.68771-8-minchan@kernel.org/

What "next" tag this (v8) submission is based on please?

> * from v6 - https://lore.kernel.org/linux-api/20200219014433.88424-1-minchan@kernel.org/
>   * fix comments and descriptions - Suren
>   * Add Reviewed-by - Suren
>   * fix build break reported by 0-day
> 
> * from v5 - https://lore.kernel.org/linux-mm/20200214170520.160271-1-minchan@kernel.org/
>   * use null task and requestor's mm for io_madvise - Jann and Jens
>   * use right commit description for moving pidfd_get_pid - Christoph
> 
> * from v4 - https://lore.kernel.org/linux-mm/20200212233946.246210-1-minchan@kernel.org/
>   * pass mm down to functions, not accessing task->mm - Jann
>   * clean up - Alexander
>   * add Reviewed-by - Alexander, SeongJae
>   * patch reordering
> 
> * from v3 - https://lore.kernel.org/linux-mm/20200128001641.5086-1-minchan@kernel.org/
>   * verify task->mm aftere access_mm - Oleg
>   * split some patches for easy review - Alexander
>   * clean up fatal signal checking - Suren
> 
> * from v2 - https://lore.kernel.org/linux-mm/20200116235953.163318-1-minchan@kernel.org/
>   * check signal callee and caller to bail out - Kirill Tkhai
>   * put more clarification for justification of new API
> 
> * from v1 - https://lore.kernel.org/linux-mm/20200110213433.94739-1-minchan@kernel.org/
>   * fix syscall number - SeongJae
>   * use get_pid_task - Kirill Tkhai
>   * extend API to support pid as well as pidfd - Kirill Tkhai
> 
> Minchan Kim (4):
>   mm/madvise: pass task and mm to do_madvise
>   pid: move pidfd_get_pid() to pid.c
>   mm/madvise: introduce process_madvise() syscall: an external memory
>     hinting API
>   mm/madvise: check fatal signal pending of target process
> 
>  arch/alpha/kernel/syscalls/syscall.tbl      |   1 +
>  arch/arm/tools/syscall.tbl                  |   1 +
>  arch/arm64/include/asm/unistd.h             |   2 +-
>  arch/arm64/include/asm/unistd32.h           |   2 +
>  arch/ia64/kernel/syscalls/syscall.tbl       |   1 +
>  arch/m68k/kernel/syscalls/syscall.tbl       |   1 +
>  arch/microblaze/kernel/syscalls/syscall.tbl |   1 +
>  arch/mips/kernel/syscalls/syscall_n32.tbl   |   1 +
>  arch/mips/kernel/syscalls/syscall_n64.tbl   |   1 +
>  arch/mips/kernel/syscalls/syscall_o32.tbl   |   1 +
>  arch/parisc/kernel/syscalls/syscall.tbl     |   1 +
>  arch/powerpc/kernel/syscalls/syscall.tbl    |   1 +
>  arch/s390/kernel/syscalls/syscall.tbl       |   1 +
>  arch/sh/kernel/syscalls/syscall.tbl         |   1 +
>  arch/sparc/kernel/syscalls/syscall.tbl      |   1 +
>  arch/x86/entry/syscalls/syscall_32.tbl      |   1 +
>  arch/x86/entry/syscalls/syscall_64.tbl      |   2 +
>  arch/xtensa/kernel/syscalls/syscall.tbl     |   1 +
>  fs/io_uring.c                               |   2 +-
>  include/linux/compat.h                      |   4 +
>  include/linux/mm.h                          |   3 +-
>  include/linux/pid.h                         |   1 +
>  include/linux/syscalls.h                    |   2 +
>  include/uapi/asm-generic/unistd.h           |   4 +-
>  kernel/exit.c                               |  17 --
>  kernel/pid.c                                |  17 ++
>  kernel/sys_ni.c                             |   2 +
>  mm/madvise.c                                | 190 +++++++++++++++++---
>  28 files changed, 217 insertions(+), 46 deletions(-)
> 
> -- 
> 2.27.0.111.gc72c7da667-goog
>
Minchan Kim June 24, 2020, 1:31 a.m. UTC | #4
Hi Oleksandr,

On Tue, Jun 23, 2020 at 11:07:21AM +0200, Oleksandr Natalenko wrote:
> On Mon, Jun 22, 2020 at 12:28:56PM -0700, Minchan Kim wrote:
> > Now, we have MADV_PAGEOUT and MADV_COLD as madvise hinting API. With that,
> > application could give hints to kernel what memory range are preferred to be
> > reclaimed. However, in some platform(e.g., Android), the information
> > required to make the hinting decision is not known to the app.
> > Instead, it is known to a centralized userspace daemon(e.g., ActivityManagerService),
> > and that daemon must be able to initiate reclaim on its own without any app
> > involvement.
> > 
> > To solve the concern, this patch introduces new syscall - process_madvise(2).
> > Bascially, it's same with madvise(2) syscall but it has some differences.
> > 
> > 1. It needs pidfd of target process to provide the hint
> > 2. It supports only MADV_{COLD|PAGEOUT} at this moment.
> >    Other hints in madvise will be opened when there are explicit requests from
> >    community to prevent unexpected bugs we couldn't support.
> > 3. Only privileged processes can do something for other process's address
> >    space.
> > 
> > For more detail of the new API, please see "mm: introduce external memory hinting API"
> > description in this patchset.
> > 
> > * from v7 -  http://lore.kernel.org/r/20200302193630.68771-1-minchan@kernel.org
> >   * dropping pid support from new syscall and fold releated patches into syscall patch
> >   * dropping KSM patch by discussion - Oleksandr, I lost the discussion.
> >     Please resend the single patch against of the patchset if you resolves the discussion.
> >     https://lore.kernel.org/linux-api/20200302193630.68771-8-minchan@kernel.org/
> 
> What "next" tag this (v8) submission is based on please?

It's against on v5.8-rc1-mmots-2020-06-20-21-44 from mmotm - https://github.com/hnaz/linux-mm.git