mm/mempolicy.c: Fix out of bounds write in mpol_parse_str()

Commit Message

Dan Carpenter Jan. 15, 2020, 5:54 a.m. UTC

What we are trying to do is change the '=' character to a NUL terminator
and then at the end of the function we restore it back to an '='.  The
problem is there are two error paths where we jump to the end of the
function before we have replaced the '=' with NUL.  We end up putting
the '=' in the wrong place (possibly one element before the start of
the buffer).

Reported-by: [e-mail]
Fixes: 095f1fc4ebf3 ("mempolicy: rework shmem mpol parsing and display")
Signed-off-by: Dan Carpenter <[e-mail]>
---
 mm/mempolicy.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

Comments

Vlastimil Babka Jan. 15, 2020, 12:54 p.m. UTC | #1

On 1/15/20 6:54 AM, Dan Carpenter wrote:
> What we are trying to do is change the '=' character to a NUL terminator
> and then at the end of the function we restore it back to an '='.  The
> problem is there are two error paths where we jump to the end of the
> function before we have replaced the '=' with NUL.  We end up putting
> the '=' in the wrong place (possibly one element before the start of
> the buffer).

Bleh.

> Reported-by: [e-mail]
> Fixes: 095f1fc4ebf3 ("mempolicy: rework shmem mpol parsing and display")
> Signed-off-by: Dan Carpenter <[e-mail]>

Acked-by: Vlastimil Babka <[e-mail]>

CC stable perhaps? Can this (tmpfs mount options parsing AFAICS?) become
part of unprivileged operation in some scenarios?

> ---
>  mm/mempolicy.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 067cf7d3daf5..1340c5c496b5 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -2817,6 +2817,9 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
>  	char *flags = strchr(str, '=');
>  	int err = 1, mode;
>  
> +	if (flags)
> +		*flags++ = '\0';	/* terminate mode string */
> +
>  	if (nodelist) {
>  		/* NUL-terminate mode or flags string */
>  		*nodelist++ = '\0';
> @@ -2827,9 +2830,6 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
>  	} else
>  		nodes_clear(nodes);
>  
> -	if (flags)
> -		*flags++ = '\0';	/* terminate mode string */
> -
>  	mode = match_string(policy_modes, MPOL_MAX, str);
>  	if (mode < 0)
>  		goto out;
>

Dmitry Vyukov Jan. 15, 2020, 12:57 p.m. UTC | #2

On Wed, Jan 15, 2020 at 1:54 PM Vlastimil Babka <[e-mail]> wrote:
>
> On 1/15/20 6:54 AM, Dan Carpenter wrote:
> > What we are trying to do is change the '=' character to a NUL terminator
> > and then at the end of the function we restore it back to an '='.  The
> > problem is there are two error paths where we jump to the end of the
> > function before we have replaced the '=' with NUL.  We end up putting
> > the '=' in the wrong place (possibly one element before the start of
> > the buffer).
>
> Bleh.
>
> > Reported-by: [e-mail]
> > Fixes: 095f1fc4ebf3 ("mempolicy: rework shmem mpol parsing and display")
> > Signed-off-by: Dan Carpenter <[e-mail]>
>
> Acked-by: Vlastimil Babka <[e-mail]>
>
> CC stable perhaps? Can this (tmpfs mount options parsing AFAICS?) become
> part of unprivileged operation in some scenarios?

Yes, tmpfs can be mounted by any user inside of a user namespace.
Also I suspect there are cases where an unprivileged attacker can
trick some utility to mount tmpfs on their behalf and provide their
own mount options.

> > ---
> >  mm/mempolicy.c | 6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> > index 067cf7d3daf5..1340c5c496b5 100644
> > --- a/mm/mempolicy.c
> > +++ b/mm/mempolicy.c
> > @@ -2817,6 +2817,9 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
> >       char *flags = strchr(str, '=');
> >       int err = 1, mode;
> >
> > +     if (flags)
> > +             *flags++ = '\0';        /* terminate mode string */
> > +
> >       if (nodelist) {
> >               /* NUL-terminate mode or flags string */
> >               *nodelist++ = '\0';
> > @@ -2827,9 +2830,6 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
> >       } else
> >               nodes_clear(nodes);
> >
> > -     if (flags)
> > -             *flags++ = '\0';        /* terminate mode string */
> > -
> >       mode = match_string(policy_modes, MPOL_MAX, str);
> >       if (mode < 0)
> >               goto out;
> >
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to [e-mail].
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/d31f6069-bda7-2cdb-b770-0c9cddac7537%40suse.cz.

Michal Hocko Jan. 15, 2020, 3:03 p.m. UTC | #3

On Wed 15-01-20 13:57:47, Dmitry Vyukov wrote:
> On Wed, Jan 15, 2020 at 1:54 PM Vlastimil Babka <[e-mail]> wrote:
> >
> > On 1/15/20 6:54 AM, Dan Carpenter wrote:
> > > What we are trying to do is change the '=' character to a NUL terminator
> > > and then at the end of the function we restore it back to an '='.  The
> > > problem is there are two error paths where we jump to the end of the
> > > function before we have replaced the '=' with NUL.  We end up putting
> > > the '=' in the wrong place (possibly one element before the start of
> > > the buffer).
> >
> > Bleh.
> >
> > > Reported-by: [e-mail]
> > > Fixes: 095f1fc4ebf3 ("mempolicy: rework shmem mpol parsing and display")
> > > Signed-off-by: Dan Carpenter <[e-mail]>
> >
> > Acked-by: Vlastimil Babka <[e-mail]>
> >
> > CC stable perhaps? Can this (tmpfs mount options parsing AFAICS?) become
> > part of unprivileged operation in some scenarios?
> 
> Yes, tmpfs can be mounted by any user inside of a user namespace.

Huh, is there any restriction though? It is certainly not nice to have
an arbitrary memory allocated without a way of reclaiming it and OOM
killer wouldn't help for shmem.

Dmitry Vyukov Jan. 15, 2020, 3:14 p.m. UTC | #4

On Wed, Jan 15, 2020 at 4:03 PM Michal Hocko <[e-mail]> wrote:
>
> On Wed 15-01-20 13:57:47, Dmitry Vyukov wrote:
> > On Wed, Jan 15, 2020 at 1:54 PM Vlastimil Babka <[e-mail]> wrote:
> > >
> > > On 1/15/20 6:54 AM, Dan Carpenter wrote:
> > > > What we are trying to do is change the '=' character to a NUL terminator
> > > > and then at the end of the function we restore it back to an '='.  The
> > > > problem is there are two error paths where we jump to the end of the
> > > > function before we have replaced the '=' with NUL.  We end up putting
> > > > the '=' in the wrong place (possibly one element before the start of
> > > > the buffer).
> > >
> > > Bleh.
> > >
> > > > Reported-by: [e-mail]
> > > > Fixes: 095f1fc4ebf3 ("mempolicy: rework shmem mpol parsing and display")
> > > > Signed-off-by: Dan Carpenter <[e-mail]>
> > >
> > > Acked-by: Vlastimil Babka <[e-mail]>
> > >
> > > CC stable perhaps? Can this (tmpfs mount options parsing AFAICS?) become
> > > part of unprivileged operation in some scenarios?
> >
> > Yes, tmpfs can be mounted by any user inside of a user namespace.
>
> Huh, is there any restriction though? It is certainly not nice to have
> an arbitrary memory allocated without a way of reclaiming it and OOM
> killer wouldn't help for shmem.

The last time I checked there were hundreds of ways to allocate
arbitrary amounts of memory without any restrictions by any user. The
example at hand was setting up GB-sized netfilter tables in netns
under userns. It's not subject to ulimit/memcg. Most kmalloc/vmalloc's
are not accounted and can be abused. Is tmpfs even worse than these?

Michal Hocko Jan. 15, 2020, 7:05 p.m. UTC | #5

On Wed 15-01-20 16:14:43, Dmitry Vyukov wrote:
> On Wed, Jan 15, 2020 at 4:03 PM Michal Hocko <[e-mail]> wrote:
> >
> > On Wed 15-01-20 13:57:47, Dmitry Vyukov wrote:
> > > On Wed, Jan 15, 2020 at 1:54 PM Vlastimil Babka <[e-mail]> wrote:
> > > >
> > > > On 1/15/20 6:54 AM, Dan Carpenter wrote:
> > > > > What we are trying to do is change the '=' character to a NUL terminator
> > > > > and then at the end of the function we restore it back to an '='.  The
> > > > > problem is there are two error paths where we jump to the end of the
> > > > > function before we have replaced the '=' with NUL.  We end up putting
> > > > > the '=' in the wrong place (possibly one element before the start of
> > > > > the buffer).
> > > >
> > > > Bleh.
> > > >
> > > > > Reported-by: [e-mail]
> > > > > Fixes: 095f1fc4ebf3 ("mempolicy: rework shmem mpol parsing and display")
> > > > > Signed-off-by: Dan Carpenter <[e-mail]>
> > > >
> > > > Acked-by: Vlastimil Babka <[e-mail]>
> > > >
> > > > CC stable perhaps? Can this (tmpfs mount options parsing AFAICS?) become
> > > > part of unprivileged operation in some scenarios?
> > >
> > > Yes, tmpfs can be mounted by any user inside of a user namespace.
> >
> > Huh, is there any restriction though? It is certainly not nice to have
> > an arbitrary memory allocated without a way of reclaiming it and OOM
> > killer wouldn't help for shmem.
> 
> The last time I checked there were hundreds of ways to allocate
> arbitrary amounts of memory without any restrictions by any user. The
> example at hand was setting up GB-sized netfilter tables in netns
> under userns. It's not subject to ulimit/memcg.

That's bad!

> Most kmalloc/vmalloc's are not accounted and can be abused.

Many of those should be bound to some objects and if those are directly
controllable by userspace then we should account at least. And if they
are not bound to a process life time then restricted.

> Is tmpfs even worse than these?

Well, tmpfs is accounted and restricted by memcg at least. The problem
is that it the memory is not really bound to a process life time which
makes it effectively unreclaimable once the swap space is depleted.
Still bad.

Dmitry Vyukov Jan. 16, 2020, 5:41 a.m. UTC | #6

On Wed, Jan 15, 2020 at 8:05 PM Michal Hocko <[e-mail]> wrote:
> > > > On Wed, Jan 15, 2020 at 1:54 PM Vlastimil Babka <[e-mail]> wrote:
> > > > >
> > > > > On 1/15/20 6:54 AM, Dan Carpenter wrote:
> > > > > > What we are trying to do is change the '=' character to a NUL terminator
> > > > > > and then at the end of the function we restore it back to an '='.  The
> > > > > > problem is there are two error paths where we jump to the end of the
> > > > > > function before we have replaced the '=' with NUL.  We end up putting
> > > > > > the '=' in the wrong place (possibly one element before the start of
> > > > > > the buffer).
> > > > >
> > > > > Bleh.
> > > > >
> > > > > > Reported-by: [e-mail]
> > > > > > Fixes: 095f1fc4ebf3 ("mempolicy: rework shmem mpol parsing and display")
> > > > > > Signed-off-by: Dan Carpenter <[e-mail]>
> > > > >
> > > > > Acked-by: Vlastimil Babka <[e-mail]>
> > > > >
> > > > > CC stable perhaps? Can this (tmpfs mount options parsing AFAICS?) become
> > > > > part of unprivileged operation in some scenarios?
> > > >
> > > > Yes, tmpfs can be mounted by any user inside of a user namespace.
> > >
> > > Huh, is there any restriction though? It is certainly not nice to have
> > > an arbitrary memory allocated without a way of reclaiming it and OOM
> > > killer wouldn't help for shmem.
> >
> > The last time I checked there were hundreds of ways to allocate
> > arbitrary amounts of memory without any restrictions by any user. The
> > example at hand was setting up GB-sized netfilter tables in netns
> > under userns. It's not subject to ulimit/memcg.
>
> That's bad!
>
> > Most kmalloc/vmalloc's are not accounted and can be abused.
>
> Many of those should be bound to some objects and if those are directly
> controllable by userspace then we should account at least. And if they
> are not bound to a process life time then restricted.

I see you actually added one GFP_ACCOUNT in netfilter in "netfilter:
x_tables: do not fail xt_alloc_table_info too easilly". But it seems
there are more:

$ grep vmalloc\( net/netfilter/*.c
net/netfilter/nf_tables_api.c: return kvmalloc(alloc, GFP_KERNEL);
net/netfilter/x_tables.c: xt[af].compat_tab = vmalloc(mem);
net/netfilter/x_tables.c: mem = vmalloc(len);
net/netfilter/x_tables.c: info = kvmalloc(sz, GFP_KERNEL_ACCOUNT);
net/netfilter/xt_hashlimit.c: /* FIXME: don't use vmalloc() here or
anywhere else -HW */
net/netfilter/xt_hashlimit.c: hinfo = vmalloc(struct_size(hinfo, hash, size));

These are not bound to processes/threads as namespaces are orthogonal to tasks.

Somebody told me that it's not good to use GFP_ACCOUNT if the
allocation is not tied to the lifetime of the process. Is it still
true?

In the end if user controls either size or number allocations, they
should be accounted, and it seems we still have thousands of
unaccounted ones. There are dozens of kmalloc's in netfilter code and
none of them use GFP_ACCOUNT...



> > Is tmpfs even worse than these?
>
> Well, tmpfs is accounted and restricted by memcg at least. The problem
> is that it the memory is not really bound to a process life time which
> makes it effectively unreclaimable once the swap space is depleted.
> Still bad.

I see. If I understand it correctly, this one is actually better than
all these non-GFP_ACCOUNT allocations.
If I would DoS a box (intentionally or unintentionally, just a bug in
my program) I would probably go for one of these easier ones without
GFP_ACCOUNT.

Michal Hocko Jan. 16, 2020, 7:39 a.m. UTC | #7

On Thu 16-01-20 06:41:46, Dmitry Vyukov wrote:
> On Wed, Jan 15, 2020 at 8:05 PM Michal Hocko <[e-mail]> wrote:
> > > > > On Wed, Jan 15, 2020 at 1:54 PM Vlastimil Babka <[e-mail]> wrote:
> > > > > >
> > > > > > On 1/15/20 6:54 AM, Dan Carpenter wrote:
> > > > > > > What we are trying to do is change the '=' character to a NUL terminator
> > > > > > > and then at the end of the function we restore it back to an '='.  The
> > > > > > > problem is there are two error paths where we jump to the end of the
> > > > > > > function before we have replaced the '=' with NUL.  We end up putting
> > > > > > > the '=' in the wrong place (possibly one element before the start of
> > > > > > > the buffer).
> > > > > >
> > > > > > Bleh.
> > > > > >
> > > > > > > Reported-by: [e-mail]
> > > > > > > Fixes: 095f1fc4ebf3 ("mempolicy: rework shmem mpol parsing and display")
> > > > > > > Signed-off-by: Dan Carpenter <[e-mail]>
> > > > > >
> > > > > > Acked-by: Vlastimil Babka <[e-mail]>
> > > > > >
> > > > > > CC stable perhaps? Can this (tmpfs mount options parsing AFAICS?) become
> > > > > > part of unprivileged operation in some scenarios?
> > > > >
> > > > > Yes, tmpfs can be mounted by any user inside of a user namespace.
> > > >
> > > > Huh, is there any restriction though? It is certainly not nice to have
> > > > an arbitrary memory allocated without a way of reclaiming it and OOM
> > > > killer wouldn't help for shmem.
> > >
> > > The last time I checked there were hundreds of ways to allocate
> > > arbitrary amounts of memory without any restrictions by any user. The
> > > example at hand was setting up GB-sized netfilter tables in netns
> > > under userns. It's not subject to ulimit/memcg.
> >
> > That's bad!
> >
> > > Most kmalloc/vmalloc's are not accounted and can be abused.
> >
> > Many of those should be bound to some objects and if those are directly
> > controllable by userspace then we should account at least. And if they
> > are not bound to a process life time then restricted.
> 
> I see you actually added one GFP_ACCOUNT in netfilter in "netfilter:
> x_tables: do not fail xt_alloc_table_info too easilly". But it seems
> there are more:
> 
> $ grep vmalloc\( net/netfilter/*.c
> net/netfilter/nf_tables_api.c: return kvmalloc(alloc, GFP_KERNEL);
> net/netfilter/x_tables.c: xt[af].compat_tab = vmalloc(mem);
> net/netfilter/x_tables.c: mem = vmalloc(len);
> net/netfilter/x_tables.c: info = kvmalloc(sz, GFP_KERNEL_ACCOUNT);
> net/netfilter/xt_hashlimit.c: /* FIXME: don't use vmalloc() here or
> anywhere else -HW */
> net/netfilter/xt_hashlimit.c: hinfo = vmalloc(struct_size(hinfo, hash, size));
> 
> These are not bound to processes/threads as namespaces are orthogonal to tasks.

I cannot really comment on those. This is for networking people to
examine and find out whether they allow an untrusted user to runaway.

> Somebody told me that it's not good to use GFP_ACCOUNT if the
> allocation is not tied to the lifetime of the process. Is it still
> true?

Those are more tricky. Mostly because there is no way to reclaim the
memory once the hard limit is hit. Even the memcg oom killer will not
help much. So a care should be taken when adding GFP_ACCOUNT for those.
On the other hand it would prevent an unbounded allocations at least
so the DoS would be reduced to the hard limited memcg.

Dmitry Vyukov Jan. 16, 2020, 10:13 a.m. UTC | #8

On Thu, Jan 16, 2020 at 8:39 AM Michal Hocko <[e-mail]> wrote:
> > > > > > > On 1/15/20 6:54 AM, Dan Carpenter wrote:
> > > > > > > > What we are trying to do is change the '=' character to a NUL terminator
> > > > > > > > and then at the end of the function we restore it back to an '='.  The
> > > > > > > > problem is there are two error paths where we jump to the end of the
> > > > > > > > function before we have replaced the '=' with NUL.  We end up putting
> > > > > > > > the '=' in the wrong place (possibly one element before the start of
> > > > > > > > the buffer).
> > > > > > >
> > > > > > > Bleh.
> > > > > > >
> > > > > > > > Reported-by: [e-mail]
> > > > > > > > Fixes: 095f1fc4ebf3 ("mempolicy: rework shmem mpol parsing and display")
> > > > > > > > Signed-off-by: Dan Carpenter <[e-mail]>
> > > > > > >
> > > > > > > Acked-by: Vlastimil Babka <[e-mail]>
> > > > > > >
> > > > > > > CC stable perhaps? Can this (tmpfs mount options parsing AFAICS?) become
> > > > > > > part of unprivileged operation in some scenarios?
> > > > > >
> > > > > > Yes, tmpfs can be mounted by any user inside of a user namespace.
> > > > >
> > > > > Huh, is there any restriction though? It is certainly not nice to have
> > > > > an arbitrary memory allocated without a way of reclaiming it and OOM
> > > > > killer wouldn't help for shmem.
> > > >
> > > > The last time I checked there were hundreds of ways to allocate
> > > > arbitrary amounts of memory without any restrictions by any user. The
> > > > example at hand was setting up GB-sized netfilter tables in netns
> > > > under userns. It's not subject to ulimit/memcg.
> > >
> > > That's bad!
> > >
> > > > Most kmalloc/vmalloc's are not accounted and can be abused.
> > >
> > > Many of those should be bound to some objects and if those are directly
> > > controllable by userspace then we should account at least. And if they
> > > are not bound to a process life time then restricted.
> >
> > I see you actually added one GFP_ACCOUNT in netfilter in "netfilter:
> > x_tables: do not fail xt_alloc_table_info too easilly". But it seems
> > there are more:
> >
> > $ grep vmalloc\( net/netfilter/*.c
> > net/netfilter/nf_tables_api.c: return kvmalloc(alloc, GFP_KERNEL);
> > net/netfilter/x_tables.c: xt[af].compat_tab = vmalloc(mem);
> > net/netfilter/x_tables.c: mem = vmalloc(len);
> > net/netfilter/x_tables.c: info = kvmalloc(sz, GFP_KERNEL_ACCOUNT);
> > net/netfilter/xt_hashlimit.c: /* FIXME: don't use vmalloc() here or
> > anywhere else -HW */
> > net/netfilter/xt_hashlimit.c: hinfo = vmalloc(struct_size(hinfo, hash, size));
> >
> > These are not bound to processes/threads as namespaces are orthogonal to tasks.
>
> I cannot really comment on those. This is for networking people to
> examine and find out whether they allow an untrusted user to runaway.

Unless I am missing an elephant in this whole picture, kernel code
contains 20K+ unaccounted allocations and if I am not mistaken few of
them were audited and are intentionally unaccounted rather than
unaccounted just because it's the default. So if we want DoS
protection, it's really for every kernel developer/maintainer to audit
and fix these allocation sites. And since we have a unikernel, a
single unaccounted allocation may compromise the whole kernel. I
assume we would need something like GFP_UNACCOUNTED to mark audited
allocations that don't need accounting and then slowly reduce number
of allocations without both ACCOUNTED and UNACCOUNTED.


> > Somebody told me that it's not good to use GFP_ACCOUNT if the
> > allocation is not tied to the lifetime of the process. Is it still
> > true?
>
> Those are more tricky. Mostly because there is no way to reclaim the
> memory once the hard limit is hit. Even the memcg oom killer will not
> help much. So a care should be taken when adding GFP_ACCOUNT for those.
> On the other hand it would prevent an unbounded allocations at least
> so the DoS would be reduced to the hard limited memcg.

What exactly is this care in practice?
It seems that in a148ce15375fc664ad64762c751c0c2aecb2cafe you just
added it and the allocation is not tied to the process. At least I
don't see any explanation as to why that one is safe, while accounting
other similar allocation is not...

Michal Hocko Jan. 16, 2020, 11:51 a.m. UTC | #9

On Thu 16-01-20 11:13:09, Dmitry Vyukov wrote:
> On Thu, Jan 16, 2020 at 8:39 AM Michal Hocko <[e-mail]> wrote:
[...]
> > > $ grep vmalloc\( net/netfilter/*.c
> > > net/netfilter/nf_tables_api.c: return kvmalloc(alloc, GFP_KERNEL);
> > > net/netfilter/x_tables.c: xt[af].compat_tab = vmalloc(mem);
> > > net/netfilter/x_tables.c: mem = vmalloc(len);
> > > net/netfilter/x_tables.c: info = kvmalloc(sz, GFP_KERNEL_ACCOUNT);
> > > net/netfilter/xt_hashlimit.c: /* FIXME: don't use vmalloc() here or
> > > anywhere else -HW */
> > > net/netfilter/xt_hashlimit.c: hinfo = vmalloc(struct_size(hinfo, hash, size));
> > >
> > > These are not bound to processes/threads as namespaces are orthogonal to tasks.
> >
> > I cannot really comment on those. This is for networking people to
> > examine and find out whether they allow an untrusted user to runaway.
> 
> Unless I am missing an elephant in this whole picture, kernel code
> contains 20K+ unaccounted allocations and if I am not mistaken few of
> them were audited and are intentionally unaccounted rather than
> unaccounted just because it's the default. So if we want DoS
> protection, it's really for every kernel developer/maintainer to audit
> and fix these allocation sites. And since we have a unikernel, a
> single unaccounted allocation may compromise the whole kernel. I
> assume we would need something like GFP_UNACCOUNTED to mark audited
> allocations that don't need accounting and then slowly reduce number
> of allocations without both ACCOUNTED and UNACCOUNTED.

This is the original approach which led to all sorts of problems and so
we switched the opt-out to opt-in. Have a look at a9bb7e620efd ("memcg:
only account kmem allocations marked as __GFP_ACCOUNT").
Our protection will never be perfect because that would require to
design the system with the protection in mind.
 
> > > Somebody told me that it's not good to use GFP_ACCOUNT if the
> > > allocation is not tied to the lifetime of the process. Is it still
> > > true?
> >
> > Those are more tricky. Mostly because there is no way to reclaim the
> > memory once the hard limit is hit. Even the memcg oom killer will not
> > help much. So a care should be taken when adding GFP_ACCOUNT for those.
> > On the other hand it would prevent an unbounded allocations at least
> > so the DoS would be reduced to the hard limited memcg.
> 
> What exactly is this care in practice?
> It seems that in a148ce15375fc664ad64762c751c0c2aecb2cafe you just
> added it and the allocation is not tied to the process. At least I
> don't see any explanation as to why that one is safe, while accounting
> other similar allocation is not...

My memory is dim but AFAIR the memcg accounting was compromise between
usability and the whole system stability. Really large tables could be
allocated by untrusted users and that was seen as a _real_ problem. The
previous solution added _some_ protection which led to regressions
even for reasonable cases though. Memcg accounting was deemed as
reasonable middle ground.

The result is that a completely depleted memcg requires an admin
intervention and the admin has to know what to do to tear it down.
Kernel cannot do anything about that. And that is the trickiness I've
had in mind. Listing page tables is something admins can do quite
easily, right? There are many other objects which are much harder to act
about. E.g. what are you going to do with tmpfs mounts? Are you going to
remove them and cause potential data loss? That being said some objects
really have to be limited even before they start consuming memory IMHO.

Dmitry Vyukov Jan. 16, 2020, 12:41 p.m. UTC | #10

On Thu, Jan 16, 2020 at 12:51 PM Michal Hocko <[e-mail]> wrote:
>
> On Thu 16-01-20 11:13:09, Dmitry Vyukov wrote:
> > On Thu, Jan 16, 2020 at 8:39 AM Michal Hocko <[e-mail]> wrote:
> [...]
> > > > $ grep vmalloc\( net/netfilter/*.c
> > > > net/netfilter/nf_tables_api.c: return kvmalloc(alloc, GFP_KERNEL);
> > > > net/netfilter/x_tables.c: xt[af].compat_tab = vmalloc(mem);
> > > > net/netfilter/x_tables.c: mem = vmalloc(len);
> > > > net/netfilter/x_tables.c: info = kvmalloc(sz, GFP_KERNEL_ACCOUNT);
> > > > net/netfilter/xt_hashlimit.c: /* FIXME: don't use vmalloc() here or
> > > > anywhere else -HW */
> > > > net/netfilter/xt_hashlimit.c: hinfo = vmalloc(struct_size(hinfo, hash, size));
> > > >
> > > > These are not bound to processes/threads as namespaces are orthogonal to tasks.
> > >
> > > I cannot really comment on those. This is for networking people to
> > > examine and find out whether they allow an untrusted user to runaway.
> >
> > Unless I am missing an elephant in this whole picture, kernel code
> > contains 20K+ unaccounted allocations and if I am not mistaken few of
> > them were audited and are intentionally unaccounted rather than
> > unaccounted just because it's the default. So if we want DoS
> > protection, it's really for every kernel developer/maintainer to audit
> > and fix these allocation sites. And since we have a unikernel, a
> > single unaccounted allocation may compromise the whole kernel. I
> > assume we would need something like GFP_UNACCOUNTED to mark audited
> > allocations that don't need accounting and then slowly reduce number
> > of allocations without both ACCOUNTED and UNACCOUNTED.
>
> This is the original approach which led to all sorts of problems and so
> we switched the opt-out to opt-in. Have a look at a9bb7e620efd ("memcg:
> only account kmem allocations marked as __GFP_ACCOUNT").
> Our protection will never be perfect because that would require to
> design the system with the protection in mind.

I don't mean to switch the default. I mean adding a way to distinguish
between reviewed and intentionally unaccounted allocation and
unreviewed allocation which is unaccounted just because that's the
default. This would allow to progress incrementally, rather than redo
the same work again and again.

> > > > Somebody told me that it's not good to use GFP_ACCOUNT if the
> > > > allocation is not tied to the lifetime of the process. Is it still
> > > > true?
> > >
> > > Those are more tricky. Mostly because there is no way to reclaim the
> > > memory once the hard limit is hit. Even the memcg oom killer will not
> > > help much. So a care should be taken when adding GFP_ACCOUNT for those.
> > > On the other hand it would prevent an unbounded allocations at least
> > > so the DoS would be reduced to the hard limited memcg.
> >
> > What exactly is this care in practice?
> > It seems that in a148ce15375fc664ad64762c751c0c2aecb2cafe you just
> > added it and the allocation is not tied to the process. At least I
> > don't see any explanation as to why that one is safe, while accounting
> > other similar allocation is not...
>
> My memory is dim but AFAIR the memcg accounting was compromise between
> usability and the whole system stability. Really large tables could be
> allocated by untrusted users and that was seen as a _real_ problem. The
> previous solution added _some_ protection which led to regressions
> even for reasonable cases though. Memcg accounting was deemed as
> reasonable middle ground.
>
> The result is that a completely depleted memcg requires an admin
> intervention and the admin has to know what to do to tear it down.
> Kernel cannot do anything about that. And that is the trickiness I've
> had in mind. Listing page tables is something admins can do quite
> easily, right? There are many other objects which are much harder to act
> about. E.g. what are you going to do with tmpfs mounts? Are you going to
> remove them and cause potential data loss? That being said some objects
> really have to be limited even before they start consuming memory IMHO.

Interesting. But there is really no admin today, or at least nothing
should rely on one in any way. Either because of the scale (you have
thousands/millions of machines and spending human time on each of them
individually is not going to fly) and/or because there is nobody
qualified enough around (e.g. who is an admin of a median android
phone? and what does they know about tearing down namespaces and
mounts) or there is nobody interested enough (it's fun sometimes, but
not always)... but I agree that retrofitting this level of resource
control into a large existing complex system is close to impossible.

Thank a lot for bearing with me and answering my questions. I have a
better understanding of this now.

Michal Hocko Jan. 16, 2020, 2:05 p.m. UTC | #11

On Thu 16-01-20 13:41:39, Dmitry Vyukov wrote:
> On Thu, Jan 16, 2020 at 12:51 PM Michal Hocko <[e-mail]> wrote:
> >
> > On Thu 16-01-20 11:13:09, Dmitry Vyukov wrote:
> > > On Thu, Jan 16, 2020 at 8:39 AM Michal Hocko <[e-mail]> wrote:
> > [...]
> > > > > $ grep vmalloc\( net/netfilter/*.c
> > > > > net/netfilter/nf_tables_api.c: return kvmalloc(alloc, GFP_KERNEL);
> > > > > net/netfilter/x_tables.c: xt[af].compat_tab = vmalloc(mem);
> > > > > net/netfilter/x_tables.c: mem = vmalloc(len);
> > > > > net/netfilter/x_tables.c: info = kvmalloc(sz, GFP_KERNEL_ACCOUNT);
> > > > > net/netfilter/xt_hashlimit.c: /* FIXME: don't use vmalloc() here or
> > > > > anywhere else -HW */
> > > > > net/netfilter/xt_hashlimit.c: hinfo = vmalloc(struct_size(hinfo, hash, size));
> > > > >
> > > > > These are not bound to processes/threads as namespaces are orthogonal to tasks.
> > > >
> > > > I cannot really comment on those. This is for networking people to
> > > > examine and find out whether they allow an untrusted user to runaway.
> > >
> > > Unless I am missing an elephant in this whole picture, kernel code
> > > contains 20K+ unaccounted allocations and if I am not mistaken few of
> > > them were audited and are intentionally unaccounted rather than
> > > unaccounted just because it's the default. So if we want DoS
> > > protection, it's really for every kernel developer/maintainer to audit
> > > and fix these allocation sites. And since we have a unikernel, a
> > > single unaccounted allocation may compromise the whole kernel. I
> > > assume we would need something like GFP_UNACCOUNTED to mark audited
> > > allocations that don't need accounting and then slowly reduce number
> > > of allocations without both ACCOUNTED and UNACCOUNTED.
> >
> > This is the original approach which led to all sorts of problems and so
> > we switched the opt-out to opt-in. Have a look at a9bb7e620efd ("memcg:
> > only account kmem allocations marked as __GFP_ACCOUNT").
> > Our protection will never be perfect because that would require to
> > design the system with the protection in mind.
> 
> I don't mean to switch the default. I mean adding a way to distinguish
> between reviewed and intentionally unaccounted allocation and
> unreviewed allocation which is unaccounted just because that's the
> default. This would allow to progress incrementally, rather than redo
> the same work again and again.

I am not really sure this would be viable just because of the sheer
number of allocations we have in the kernel. But I would be more than
happy to be proven wrong ;)

> > > > > Somebody told me that it's not good to use GFP_ACCOUNT if the
> > > > > allocation is not tied to the lifetime of the process. Is it still
> > > > > true?
> > > >
> > > > Those are more tricky. Mostly because there is no way to reclaim the
> > > > memory once the hard limit is hit. Even the memcg oom killer will not
> > > > help much. So a care should be taken when adding GFP_ACCOUNT for those.
> > > > On the other hand it would prevent an unbounded allocations at least
> > > > so the DoS would be reduced to the hard limited memcg.
> > >
> > > What exactly is this care in practice?
> > > It seems that in a148ce15375fc664ad64762c751c0c2aecb2cafe you just
> > > added it and the allocation is not tied to the process. At least I
> > > don't see any explanation as to why that one is safe, while accounting
> > > other similar allocation is not...
> >
> > My memory is dim but AFAIR the memcg accounting was compromise between
> > usability and the whole system stability. Really large tables could be
> > allocated by untrusted users and that was seen as a _real_ problem. The
> > previous solution added _some_ protection which led to regressions
> > even for reasonable cases though. Memcg accounting was deemed as
> > reasonable middle ground.
> >
> > The result is that a completely depleted memcg requires an admin
> > intervention and the admin has to know what to do to tear it down.
> > Kernel cannot do anything about that. And that is the trickiness I've
> > had in mind. Listing page tables is something admins can do quite
> > easily, right? There are many other objects which are much harder to act
> > about. E.g. what are you going to do with tmpfs mounts? Are you going to
> > remove them and cause potential data loss? That being said some objects
> > really have to be limited even before they start consuming memory IMHO.
> 
> Interesting. But there is really no admin today, or at least nothing
> should rely on one in any way. Either because of the scale (you have
> thousands/millions of machines and spending human time on each of them
> individually is not going to fly) and/or because there is nobody
> qualified enough around (e.g. who is an admin of a median android
> phone? and what does they know about tearing down namespaces and
> mounts) or there is nobody interested enough (it's fun sometimes, but
> not always)...

If there is nobody in control then there should be a reasonably safe
policy defined at least. This is what I mentioned earlier when saying
that not all objects can be easily accounted. There has to be a strategy
defined for corner cases. And user namespaces seem to really beg for
that when you are allowed to control resources like tmpfs and others
alike.

> but I agree that retrofitting this level of resource
> control into a large existing complex system is close to impossible.
> 
> Thank a lot for bearing with me and answering my questions. I have a
> better understanding of this now.

I am glad I could help.

Thanks for giving some scary examples ;)

Patch

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 067cf7d3daf5..1340c5c496b5 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2817,6 +2817,9 @@  int mpol_parse_str(char *str, struct mempolicy **mpol)
 	char *flags = strchr(str, '=');
 	int err = 1, mode;
 
+	if (flags)
+		*flags++ = '\0';	/* terminate mode string */
+
 	if (nodelist) {
 		/* NUL-terminate mode or flags string */
 		*nodelist++ = '\0';
@@ -2827,9 +2830,6 @@  int mpol_parse_str(char *str, struct mempolicy **mpol)
 	} else
 		nodes_clear(nodes);
 
-	if (flags)
-		*flags++ = '\0';	/* terminate mode string */
-
 	mode = match_string(policy_modes, MPOL_MAX, str);
 	if (mode < 0)
 		goto out;