| Message ID | 20210818082702.692117-1-y-goto@fujitsu.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Delegated to: | Jason Gunthorpe |
| Headers | show |
| Series | RDMA/core: EPERM should be returned when # of pined pages is over ulimit | expand |
On Wed, Aug 18, 2021 at 05:27:02PM +0900, Yasunori Goto wrote: > Hello, > > When I started to use SoftRoCE, I'm very confused by > ENOMEM error output even if I gave enough memory. > > I think EPERM is more suitable for uses to solve error rather than > ENOMEM at here of ib_umem_get() when # of pinned pages is over ulimit. > This is not "memory is not enough" problem, because driver can > succeed to pin enough amount of pages, but it is larger than ulimit value. > > The hard limit of "max locked memory" can be changed by limit.conf. > In addition, this checks also CAP_IPC_LOCK, it is indeed permmission check. > So, I think the following patch. > > If there is a intention why ENOMEM is used here, please let me know. > Otherwise, I'm glad if this is merged. > > Thanks. > > > --- > When # of pinned pages are larger than ulimit of "max locked memory" > without CAP_IPC_LOCK, current ib_umem_get() returns ENOMEM. > But it does not mean "not enough memory", because driver could succeed to > pinned enough pages. > This is just capability error. Even if a normal user is limited > his/her # of pinned pages, system administrator can give permission > by change hard limit of this ulimit value. > To notify correct information to user, ib_umem_get() > should return EPERM instead of ENOMEM at here. I'm not convinced, can you find other places checking the ulimit and list what codes they return? Jason
On 2021/08/20 8:10, Jason Gunthorpe wrote: > On Wed, Aug 18, 2021 at 05:27:02PM +0900, Yasunori Goto wrote: >> Hello, >> >> When I started to use SoftRoCE, I'm very confused by >> ENOMEM error output even if I gave enough memory. >> >> I think EPERM is more suitable for uses to solve error rather than >> ENOMEM at here of ib_umem_get() when # of pinned pages is over ulimit. >> This is not "memory is not enough" problem, because driver can >> succeed to pin enough amount of pages, but it is larger than ulimit value. >> >> The hard limit of "max locked memory" can be changed by limit.conf. >> In addition, this checks also CAP_IPC_LOCK, it is indeed permmission check. >> So, I think the following patch. >> >> If there is a intention why ENOMEM is used here, please let me know. >> Otherwise, I'm glad if this is merged. >> >> Thanks. >> >> >> --- >> When # of pinned pages are larger than ulimit of "max locked memory" >> without CAP_IPC_LOCK, current ib_umem_get() returns ENOMEM. >> But it does not mean "not enough memory", because driver could succeed to >> pinned enough pages. >> This is just capability error. Even if a normal user is limited >> his/her # of pinned pages, system administrator can give permission >> by change hard limit of this ulimit value. >> To notify correct information to user, ib_umem_get() >> should return EPERM instead of ENOMEM at here. > > I'm not convinced, can you find other places checking the ulimit and > list what codes they return? Hmm, OK. I'll investigate it.
On 2021/08/20 9:36, Yasunori Goto wrote: > > > On 2021/08/20 8:10, Jason Gunthorpe wrote: >> On Wed, Aug 18, 2021 at 05:27:02PM +0900, Yasunori Goto wrote: >>> Hello, >>> >>> When I started to use SoftRoCE, I'm very confused by >>> ENOMEM error output even if I gave enough memory. >>> >>> I think EPERM is more suitable for uses to solve error rather than >>> ENOMEM at here of ib_umem_get() when # of pinned pages is over ulimit. >>> This is not "memory is not enough" problem, because driver can >>> succeed to pin enough amount of pages, but it is larger than ulimit >>> value. >>> >>> The hard limit of "max locked memory" can be changed by limit.conf. >>> In addition, this checks also CAP_IPC_LOCK, it is indeed permmission >>> check. >>> So, I think the following patch. >>> >>> If there is a intention why ENOMEM is used here, please let me know. >>> Otherwise, I'm glad if this is merged. >>> >>> Thanks. >>> >>> >>> --- >>> When # of pinned pages are larger than ulimit of "max locked memory" >>> without CAP_IPC_LOCK, current ib_umem_get() returns ENOMEM. >>> But it does not mean "not enough memory", because driver could >>> succeed to >>> pinned enough pages. >>> This is just capability error. Even if a normal user is limited >>> his/her # of pinned pages, system administrator can give permission >>> by change hard limit of this ulimit value. >>> To notify correct information to user, ib_umem_get() >>> should return EPERM instead of ENOMEM at here. >> >> I'm not convinced, can you find other places checking the ulimit and >> list what codes they return? > > Hmm, OK. > > I'll investigate it. After the investigation, I found the followings. - Many codes return ENOMEM in kernel/driver. - Only one exception I could find is perf_mmap() in kernel/events/core.c It returns EPERM. ---- static int perf_mmap(struct file *file, struct vm_area_struct *vma) { : : lock_limit = rlimit(RLIMIT_MEMLOCK); lock_limit >>= PAGE_SHIFT; locked = atomic64_read(&vma->vm_mm->pinned_vm) + extra; if ((locked > lock_limit) && perf_is_paranoid() && !capable(CAP_IPC_LOCK)) { ret = -EPERM; <----!!! goto unlock; } ---- - The man pages of mlock(2) says the followings. This seems to be cause why ENOMEM is returned in many place. ---- ENOMEM (Linux 2.6.9 and later) the caller had a nonzero RLIMIT_MEMLOCK soft resource limit, but tried to lock more memory than the limit permitted. This limit is not enforced if the process is privileged (CAP_IPC_LOCK). --- - In addition, POSIX specification(*) also says the followings at mlock(2). --- [ENOMEM] Locking the pages mapped by the specified range would exceed an implementation-defined limit on the amount of memory that the process may lock. ---- (*) https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/ So, I changed my mind now. ib_umem_get() should return ENOMEM. However, I want to provide some information to make it easy for users to understand. For example, sev_pin_memory() of arch/x86/kvm/svm/sev.c outputs error message like the followings. --- static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, : : if (locked > lock_limit && !capable(CAP_IPC_LOCK)) { pr_err("SEV: %lu locked pages exceed the lock limit of %lu.\n", locked, lock_limit); return ERR_PTR(-ENOMEM); } --- I think it is better than nothing. How do you think? Thanks, -- - Yasunori Goto
On Fri, Aug 20, 2021 at 05:45:54PM +0900, Yasunori Goto wrote: > static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, > : > : > if (locked > lock_limit && !capable(CAP_IPC_LOCK)) { > pr_err("SEV: %lu locked pages exceed the lock limit of > %lu.\n", locked, lock_limit); > return ERR_PTR(-ENOMEM); > } > > I think it is better than nothing. How do you think? Unprivileged user space should not be allowed to cause the kernel to print messages. Jason
On 2021/08/26 22:32, Jason Gunthorpe wrote: > On Fri, Aug 20, 2021 at 05:45:54PM +0900, Yasunori Goto wrote: > >> static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, >> : >> : >> if (locked > lock_limit && !capable(CAP_IPC_LOCK)) { >> pr_err("SEV: %lu locked pages exceed the lock limit of >> %lu.\n", locked, lock_limit); >> return ERR_PTR(-ENOMEM); >> } >> >> I think it is better than nothing. How do you think? > > Unprivileged user space should not be allowed to cause the kernel to > print messages. Hmm... Ok. I see. Thank you for your answer! Bye, --- Yasunori Goto
diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index 0eb40025075f..9771134649e9 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -205,7 +205,7 @@ struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, new_pinned = atomic64_add_return(npages, &mm->pinned_vm); if (new_pinned > lock_limit && !capable(CAP_IPC_LOCK)) { atomic64_sub(npages, &mm->pinned_vm); - ret = -ENOMEM; + ret = -EPERM; goto out; }
Hello, When I started to use SoftRoCE, I'm very confused by ENOMEM error output even if I gave enough memory. I think EPERM is more suitable for uses to solve error rather than ENOMEM at here of ib_umem_get() when # of pinned pages is over ulimit. This is not "memory is not enough" problem, because driver can succeed to pin enough amount of pages, but it is larger than ulimit value. The hard limit of "max locked memory" can be changed by limit.conf. In addition, this checks also CAP_IPC_LOCK, it is indeed permmission check. So, I think the following patch. If there is a intention why ENOMEM is used here, please let me know. Otherwise, I'm glad if this is merged. Thanks. --- When # of pinned pages are larger than ulimit of "max locked memory" without CAP_IPC_LOCK, current ib_umem_get() returns ENOMEM. But it does not mean "not enough memory", because driver could succeed to pinned enough pages. This is just capability error. Even if a normal user is limited his/her # of pinned pages, system administrator can give permission by change hard limit of this ulimit value. To notify correct information to user, ib_umem_get() should return EPERM instead of ENOMEM at here. Signed-off-by: Yasunori Goto <y-goto@fujitsu.com> --- drivers/infiniband/core/umem.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)