[v3] mm: fix race between kmem_cache destroy, create and deactivate
diff mbox

Message ID 20180530001204.183758-1-shakeelb@google.com
State New
Headers show

Commit Message

Shakeel Butt May 30, 2018, 12:12 a.m. UTC
The memcg kmem cache creation and deactivation (SLUB only) is
asynchronous. If a root kmem cache is destroyed whose memcg cache is in
the process of creation or deactivation, the kernel may crash.

Example of one such crash:
	general protection fault: 0000 [#1] SMP PTI
	CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp
	...
	Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn
	RIP: 0010:has_cpu_slab
	...
	Call Trace:
	? on_each_cpu_cond
	__kmem_cache_shrink
	kmemcg_cache_deact_after_rcu
	kmemcg_deactivate_workfn
	process_one_work
	worker_thread
	kthread
	ret_from_fork+0x35/0x40

To fix this race, on root kmem cache destruction, mark the cache as
dying and flush the workqueue used for memcg kmem cache creation and
deactivation.

Signed-off-by: Shakeel Butt <shakeelb@google.com>
---
Changelog since v2:
- Instead of refcount, flush the workqueue

Changelog since v1:
- Added more documentation to the code
- Renamed fields to be more readable

---
 include/linux/slab.h |  1 +
 mm/slab_common.c     | 21 ++++++++++++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)

Comments

Vladimir Davydov June 9, 2018, 10:20 a.m. UTC | #1
On Tue, May 29, 2018 at 05:12:04PM -0700, Shakeel Butt wrote:
> The memcg kmem cache creation and deactivation (SLUB only) is
> asynchronous. If a root kmem cache is destroyed whose memcg cache is in
> the process of creation or deactivation, the kernel may crash.
> 
> Example of one such crash:
> 	general protection fault: 0000 [#1] SMP PTI
> 	CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp
> 	...
> 	Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn
> 	RIP: 0010:has_cpu_slab
> 	...
> 	Call Trace:
> 	? on_each_cpu_cond
> 	__kmem_cache_shrink
> 	kmemcg_cache_deact_after_rcu
> 	kmemcg_deactivate_workfn
> 	process_one_work
> 	worker_thread
> 	kthread
> 	ret_from_fork+0x35/0x40
> 
> To fix this race, on root kmem cache destruction, mark the cache as
> dying and flush the workqueue used for memcg kmem cache creation and
> deactivation.

> @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s)
>  	if (unlikely(!s))
>  		return;
>  
> +	flush_memcg_workqueue(s);
> +

This should definitely help against async memcg_kmem_cache_create(),
but I'm afraid it doesn't eliminate the race with async destruction,
unfortunately, because the latter uses call_rcu_sched():

  memcg_deactivate_kmem_caches
   __kmem_cache_deactivate
    slab_deactivate_memcg_cache_rcu_sched
     call_rcu_sched
                                            kmem_cache_destroy
                                             shutdown_memcg_caches
                                              shutdown_cache
      memcg_deactivate_rcufn
       <dereference destroyed cache>

Can we somehow flush those pending rcu requests?
Shakeel Butt June 10, 2018, 2:52 p.m. UTC | #2
On Sat, Jun 9, 2018 at 3:20 AM Vladimir Davydov <vdavydov.dev@gmail.com> wrote:
>
> On Tue, May 29, 2018 at 05:12:04PM -0700, Shakeel Butt wrote:
> > The memcg kmem cache creation and deactivation (SLUB only) is
> > asynchronous. If a root kmem cache is destroyed whose memcg cache is in
> > the process of creation or deactivation, the kernel may crash.
> >
> > Example of one such crash:
> >       general protection fault: 0000 [#1] SMP PTI
> >       CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp
> >       ...
> >       Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn
> >       RIP: 0010:has_cpu_slab
> >       ...
> >       Call Trace:
> >       ? on_each_cpu_cond
> >       __kmem_cache_shrink
> >       kmemcg_cache_deact_after_rcu
> >       kmemcg_deactivate_workfn
> >       process_one_work
> >       worker_thread
> >       kthread
> >       ret_from_fork+0x35/0x40
> >
> > To fix this race, on root kmem cache destruction, mark the cache as
> > dying and flush the workqueue used for memcg kmem cache creation and
> > deactivation.
>
> > @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s)
> >       if (unlikely(!s))
> >               return;
> >
> > +     flush_memcg_workqueue(s);
> > +
>
> This should definitely help against async memcg_kmem_cache_create(),
> but I'm afraid it doesn't eliminate the race with async destruction,
> unfortunately, because the latter uses call_rcu_sched():
>
>   memcg_deactivate_kmem_caches
>    __kmem_cache_deactivate
>     slab_deactivate_memcg_cache_rcu_sched
>      call_rcu_sched
>                                             kmem_cache_destroy
>                                              shutdown_memcg_caches
>                                               shutdown_cache
>       memcg_deactivate_rcufn
>        <dereference destroyed cache>
>
> Can we somehow flush those pending rcu requests?

You are right and thanks for catching that. Now I am wondering if
synchronize_sched() just before flush_workqueue() should be enough.
Otherwise we might have to replace call_sched_rcu with
synchronize_sched() in kmemcg_deactivate_workfn which I would not
prefer as that would holdup the kmem_cache workqueue.

+Paul

Paul, we have a situation something similar to the following pseudo code.

CPU0:
lock(l)
if (!flag)
  call_rcu_sched(callback);
unlock(l)
------
CPU1:
lock(l)
flag = true
unlock(l)
synchronize_sched()
------

If CPU0 has called already called call_rchu_sched(callback) then later
if CPU1 calls synchronize_sched(). Is there any guarantee that on
return from synchronize_sched(), the rcu callback scheduled by CPU0
has already been executed?

thanks,
Shakeel
Paul E. McKenney June 10, 2018, 4:34 p.m. UTC | #3
On Sun, Jun 10, 2018 at 07:52:50AM -0700, Shakeel Butt wrote:
> On Sat, Jun 9, 2018 at 3:20 AM Vladimir Davydov <vdavydov.dev@gmail.com> wrote:
> >
> > On Tue, May 29, 2018 at 05:12:04PM -0700, Shakeel Butt wrote:
> > > The memcg kmem cache creation and deactivation (SLUB only) is
> > > asynchronous. If a root kmem cache is destroyed whose memcg cache is in
> > > the process of creation or deactivation, the kernel may crash.
> > >
> > > Example of one such crash:
> > >       general protection fault: 0000 [#1] SMP PTI
> > >       CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp
> > >       ...
> > >       Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn
> > >       RIP: 0010:has_cpu_slab
> > >       ...
> > >       Call Trace:
> > >       ? on_each_cpu_cond
> > >       __kmem_cache_shrink
> > >       kmemcg_cache_deact_after_rcu
> > >       kmemcg_deactivate_workfn
> > >       process_one_work
> > >       worker_thread
> > >       kthread
> > >       ret_from_fork+0x35/0x40
> > >
> > > To fix this race, on root kmem cache destruction, mark the cache as
> > > dying and flush the workqueue used for memcg kmem cache creation and
> > > deactivation.
> >
> > > @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s)
> > >       if (unlikely(!s))
> > >               return;
> > >
> > > +     flush_memcg_workqueue(s);
> > > +
> >
> > This should definitely help against async memcg_kmem_cache_create(),
> > but I'm afraid it doesn't eliminate the race with async destruction,
> > unfortunately, because the latter uses call_rcu_sched():
> >
> >   memcg_deactivate_kmem_caches
> >    __kmem_cache_deactivate
> >     slab_deactivate_memcg_cache_rcu_sched
> >      call_rcu_sched
> >                                             kmem_cache_destroy
> >                                              shutdown_memcg_caches
> >                                               shutdown_cache
> >       memcg_deactivate_rcufn
> >        <dereference destroyed cache>
> >
> > Can we somehow flush those pending rcu requests?
> 
> You are right and thanks for catching that. Now I am wondering if
> synchronize_sched() just before flush_workqueue() should be enough.
> Otherwise we might have to replace call_sched_rcu with
> synchronize_sched() in kmemcg_deactivate_workfn which I would not
> prefer as that would holdup the kmem_cache workqueue.
> 
> +Paul
> 
> Paul, we have a situation something similar to the following pseudo code.
> 
> CPU0:
> lock(l)
> if (!flag)
>   call_rcu_sched(callback);
> unlock(l)
> ------
> CPU1:
> lock(l)
> flag = true
> unlock(l)
> synchronize_sched()
> ------
> 
> If CPU0 has called already called call_rchu_sched(callback) then later
> if CPU1 calls synchronize_sched(). Is there any guarantee that on
> return from synchronize_sched(), the rcu callback scheduled by CPU0
> has already been executed?

No.  There is no such guarantee.

You instead want rcu_barrier_sched(), which waits for the callbacks from
all prior invocations of call_rcu_sched() to be invoked.

Please note that synchronize_sched() is -not- sufficient.  It is only
guaranteed to wait for a grace period, not necessarily for all prior
callbacks.  This goes both directions because if there are no callbacks
in the system, then rcu_barrier_sched() is within its rights to return
immediately.

So please make sure you use each of synchronize_sched() and
rcu_barrier_sched() to do the job that it was intended to do!  ;-)

If your lock(l) is shorthand for spin_lock(&l), it looks to me like you
actually only need rcu_barrier_sched():

	CPU0:
	spin_lock(&l);
	if (!flag)
	  call_rcu_sched(callback);
	spin_unlock(&l);

	CPU1:
	spin_lock(&l);
	flag = true;
	spin_unlock(&l);
	/* At this point, no more callbacks will be registered. */
	rcu_barrier_sched();
	/* At this point, all registered callbacks will have been invoked. */

On the other hand, if your "lock(l)" was instead shorthand for
rcu_read_lock_sched(), then you need -both- synchronize_sched() -and-
rcu_barrier().  And even then, you will be broken in -rt kernels.
(Which might or might not be a concern, depending on whether your code
matters to -rt kernels.

Make sense?

							Thanx, Paul
Shakeel Butt June 10, 2018, 5:40 p.m. UTC | #4
On Sun, Jun 10, 2018 at 9:32 AM Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
>
> On Sun, Jun 10, 2018 at 07:52:50AM -0700, Shakeel Butt wrote:
> > On Sat, Jun 9, 2018 at 3:20 AM Vladimir Davydov <vdavydov.dev@gmail.com> wrote:
> > >
> > > On Tue, May 29, 2018 at 05:12:04PM -0700, Shakeel Butt wrote:
> > > > The memcg kmem cache creation and deactivation (SLUB only) is
> > > > asynchronous. If a root kmem cache is destroyed whose memcg cache is in
> > > > the process of creation or deactivation, the kernel may crash.
> > > >
> > > > Example of one such crash:
> > > >       general protection fault: 0000 [#1] SMP PTI
> > > >       CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp
> > > >       ...
> > > >       Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn
> > > >       RIP: 0010:has_cpu_slab
> > > >       ...
> > > >       Call Trace:
> > > >       ? on_each_cpu_cond
> > > >       __kmem_cache_shrink
> > > >       kmemcg_cache_deact_after_rcu
> > > >       kmemcg_deactivate_workfn
> > > >       process_one_work
> > > >       worker_thread
> > > >       kthread
> > > >       ret_from_fork+0x35/0x40
> > > >
> > > > To fix this race, on root kmem cache destruction, mark the cache as
> > > > dying and flush the workqueue used for memcg kmem cache creation and
> > > > deactivation.
> > >
> > > > @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s)
> > > >       if (unlikely(!s))
> > > >               return;
> > > >
> > > > +     flush_memcg_workqueue(s);
> > > > +
> > >
> > > This should definitely help against async memcg_kmem_cache_create(),
> > > but I'm afraid it doesn't eliminate the race with async destruction,
> > > unfortunately, because the latter uses call_rcu_sched():
> > >
> > >   memcg_deactivate_kmem_caches
> > >    __kmem_cache_deactivate
> > >     slab_deactivate_memcg_cache_rcu_sched
> > >      call_rcu_sched
> > >                                             kmem_cache_destroy
> > >                                              shutdown_memcg_caches
> > >                                               shutdown_cache
> > >       memcg_deactivate_rcufn
> > >        <dereference destroyed cache>
> > >
> > > Can we somehow flush those pending rcu requests?
> >
> > You are right and thanks for catching that. Now I am wondering if
> > synchronize_sched() just before flush_workqueue() should be enough.
> > Otherwise we might have to replace call_sched_rcu with
> > synchronize_sched() in kmemcg_deactivate_workfn which I would not
> > prefer as that would holdup the kmem_cache workqueue.
> >
> > +Paul
> >
> > Paul, we have a situation something similar to the following pseudo code.
> >
> > CPU0:
> > lock(l)
> > if (!flag)
> >   call_rcu_sched(callback);
> > unlock(l)
> > ------
> > CPU1:
> > lock(l)
> > flag = true
> > unlock(l)
> > synchronize_sched()
> > ------
> >
> > If CPU0 has called already called call_rchu_sched(callback) then later
> > if CPU1 calls synchronize_sched(). Is there any guarantee that on
> > return from synchronize_sched(), the rcu callback scheduled by CPU0
> > has already been executed?
>
> No.  There is no such guarantee.
>
> You instead want rcu_barrier_sched(), which waits for the callbacks from
> all prior invocations of call_rcu_sched() to be invoked.
>
> Please note that synchronize_sched() is -not- sufficient.  It is only
> guaranteed to wait for a grace period, not necessarily for all prior
> callbacks.  This goes both directions because if there are no callbacks
> in the system, then rcu_barrier_sched() is within its rights to return
> immediately.
>
> So please make sure you use each of synchronize_sched() and
> rcu_barrier_sched() to do the job that it was intended to do!  ;-)
>
> If your lock(l) is shorthand for spin_lock(&l), it looks to me like you
> actually only need rcu_barrier_sched():
>
>         CPU0:
>         spin_lock(&l);
>         if (!flag)
>           call_rcu_sched(callback);
>         spin_unlock(&l);
>
>         CPU1:
>         spin_lock(&l);
>         flag = true;
>         spin_unlock(&l);
>         /* At this point, no more callbacks will be registered. */
>         rcu_barrier_sched();
>         /* At this point, all registered callbacks will have been invoked. */
>
> On the other hand, if your "lock(l)" was instead shorthand for
> rcu_read_lock_sched(), then you need -both- synchronize_sched() -and-
> rcu_barrier().  And even then, you will be broken in -rt kernels.
> (Which might or might not be a concern, depending on whether your code
> matters to -rt kernels.
>
> Make sense?
>

Thanks a lot, that was really helpful. The lock is actually
mutex_lock. So, I think rcu_barrier_sched() should be sufficient.

Shakeel
Paul E. McKenney June 10, 2018, 11:59 p.m. UTC | #5
On Sun, Jun 10, 2018 at 10:40:17AM -0700, Shakeel Butt wrote:
> On Sun, Jun 10, 2018 at 9:32 AM Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> >
> > On Sun, Jun 10, 2018 at 07:52:50AM -0700, Shakeel Butt wrote:
> > > On Sat, Jun 9, 2018 at 3:20 AM Vladimir Davydov <vdavydov.dev@gmail.com> wrote:
> > > >
> > > > On Tue, May 29, 2018 at 05:12:04PM -0700, Shakeel Butt wrote:
> > > > > The memcg kmem cache creation and deactivation (SLUB only) is
> > > > > asynchronous. If a root kmem cache is destroyed whose memcg cache is in
> > > > > the process of creation or deactivation, the kernel may crash.
> > > > >
> > > > > Example of one such crash:
> > > > >       general protection fault: 0000 [#1] SMP PTI
> > > > >       CPU: 1 PID: 1721 Comm: kworker/14:1 Not tainted 4.17.0-smp
> > > > >       ...
> > > > >       Workqueue: memcg_kmem_cache kmemcg_deactivate_workfn
> > > > >       RIP: 0010:has_cpu_slab
> > > > >       ...
> > > > >       Call Trace:
> > > > >       ? on_each_cpu_cond
> > > > >       __kmem_cache_shrink
> > > > >       kmemcg_cache_deact_after_rcu
> > > > >       kmemcg_deactivate_workfn
> > > > >       process_one_work
> > > > >       worker_thread
> > > > >       kthread
> > > > >       ret_from_fork+0x35/0x40
> > > > >
> > > > > To fix this race, on root kmem cache destruction, mark the cache as
> > > > > dying and flush the workqueue used for memcg kmem cache creation and
> > > > > deactivation.
> > > >
> > > > > @@ -845,6 +862,8 @@ void kmem_cache_destroy(struct kmem_cache *s)
> > > > >       if (unlikely(!s))
> > > > >               return;
> > > > >
> > > > > +     flush_memcg_workqueue(s);
> > > > > +
> > > >
> > > > This should definitely help against async memcg_kmem_cache_create(),
> > > > but I'm afraid it doesn't eliminate the race with async destruction,
> > > > unfortunately, because the latter uses call_rcu_sched():
> > > >
> > > >   memcg_deactivate_kmem_caches
> > > >    __kmem_cache_deactivate
> > > >     slab_deactivate_memcg_cache_rcu_sched
> > > >      call_rcu_sched
> > > >                                             kmem_cache_destroy
> > > >                                              shutdown_memcg_caches
> > > >                                               shutdown_cache
> > > >       memcg_deactivate_rcufn
> > > >        <dereference destroyed cache>
> > > >
> > > > Can we somehow flush those pending rcu requests?
> > >
> > > You are right and thanks for catching that. Now I am wondering if
> > > synchronize_sched() just before flush_workqueue() should be enough.
> > > Otherwise we might have to replace call_sched_rcu with
> > > synchronize_sched() in kmemcg_deactivate_workfn which I would not
> > > prefer as that would holdup the kmem_cache workqueue.
> > >
> > > +Paul
> > >
> > > Paul, we have a situation something similar to the following pseudo code.
> > >
> > > CPU0:
> > > lock(l)
> > > if (!flag)
> > >   call_rcu_sched(callback);
> > > unlock(l)
> > > ------
> > > CPU1:
> > > lock(l)
> > > flag = true
> > > unlock(l)
> > > synchronize_sched()
> > > ------
> > >
> > > If CPU0 has called already called call_rchu_sched(callback) then later
> > > if CPU1 calls synchronize_sched(). Is there any guarantee that on
> > > return from synchronize_sched(), the rcu callback scheduled by CPU0
> > > has already been executed?
> >
> > No.  There is no such guarantee.
> >
> > You instead want rcu_barrier_sched(), which waits for the callbacks from
> > all prior invocations of call_rcu_sched() to be invoked.
> >
> > Please note that synchronize_sched() is -not- sufficient.  It is only
> > guaranteed to wait for a grace period, not necessarily for all prior
> > callbacks.  This goes both directions because if there are no callbacks
> > in the system, then rcu_barrier_sched() is within its rights to return
> > immediately.
> >
> > So please make sure you use each of synchronize_sched() and
> > rcu_barrier_sched() to do the job that it was intended to do!  ;-)
> >
> > If your lock(l) is shorthand for spin_lock(&l), it looks to me like you
> > actually only need rcu_barrier_sched():
> >
> >         CPU0:
> >         spin_lock(&l);
> >         if (!flag)
> >           call_rcu_sched(callback);
> >         spin_unlock(&l);
> >
> >         CPU1:
> >         spin_lock(&l);
> >         flag = true;
> >         spin_unlock(&l);
> >         /* At this point, no more callbacks will be registered. */
> >         rcu_barrier_sched();
> >         /* At this point, all registered callbacks will have been invoked. */
> >
> > On the other hand, if your "lock(l)" was instead shorthand for
> > rcu_read_lock_sched(), then you need -both- synchronize_sched() -and-
> > rcu_barrier().  And even then, you will be broken in -rt kernels.
> > (Which might or might not be a concern, depending on whether your code
> > matters to -rt kernels.
> >
> > Make sense?
> 
> Thanks a lot, that was really helpful. The lock is actually
> mutex_lock. So, I think rcu_barrier_sched() should be sufficient.

Yes, with either spin_lock() or mutex_lock(), this should work.  Mutual
exclusion and all that.  ;-)

							Thanx, Paul

Patch
diff mbox

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 9ebe659bd4a5..71c5467d99c1 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -658,6 +658,7 @@  struct memcg_cache_params {
 			struct memcg_cache_array __rcu *memcg_caches;
 			struct list_head __root_caches_node;
 			struct list_head children;
+			bool dying;
 		};
 		struct {
 			struct mem_cgroup *memcg;
diff --git a/mm/slab_common.c b/mm/slab_common.c
index b0dd9db1eb2f..a3496375f5ea 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -136,6 +136,7 @@  void slab_init_memcg_params(struct kmem_cache *s)
 	s->memcg_params.root_cache = NULL;
 	RCU_INIT_POINTER(s->memcg_params.memcg_caches, NULL);
 	INIT_LIST_HEAD(&s->memcg_params.children);
+	s->memcg_params.dying = false;
 }
 
 static int init_memcg_params(struct kmem_cache *s,
@@ -608,7 +609,7 @@  void memcg_create_kmem_cache(struct mem_cgroup *memcg,
 	 * The memory cgroup could have been offlined while the cache
 	 * creation work was pending.
 	 */
-	if (memcg->kmem_state != KMEM_ONLINE)
+	if (memcg->kmem_state != KMEM_ONLINE || root_cache->memcg_params.dying)
 		goto out_unlock;
 
 	idx = memcg_cache_id(memcg);
@@ -712,6 +713,9 @@  void slab_deactivate_memcg_cache_rcu_sched(struct kmem_cache *s,
 	    WARN_ON_ONCE(s->memcg_params.deact_fn))
 		return;
 
+	if (s->memcg_params.root_cache->memcg_params.dying)
+		return;
+
 	/* pin memcg so that @s doesn't get destroyed in the middle */
 	css_get(&s->memcg_params.memcg->css);
 
@@ -823,11 +827,24 @@  static int shutdown_memcg_caches(struct kmem_cache *s)
 		return -EBUSY;
 	return 0;
 }
+
+static void flush_memcg_workqueue(struct kmem_cache *s)
+{
+	mutex_lock(&slab_mutex);
+	s->memcg_params.dying = true;
+	mutex_unlock(&slab_mutex);
+
+	flush_workqueue(memcg_kmem_cache_wq);
+}
 #else
 static inline int shutdown_memcg_caches(struct kmem_cache *s)
 {
 	return 0;
 }
+
+static inline void flush_memcg_workqueue(struct kmem_cache *s)
+{
+}
 #endif /* CONFIG_MEMCG && !CONFIG_SLOB */
 
 void slab_kmem_cache_release(struct kmem_cache *s)
@@ -845,6 +862,8 @@  void kmem_cache_destroy(struct kmem_cache *s)
 	if (unlikely(!s))
 		return;
 
+	flush_memcg_workqueue(s);
+
 	get_online_cpus();
 	get_online_mems();