[v5] mm/memory_hotplug: Fix remove_memory() lockdep splat
diff mbox series

Message ID 157991441887.2763922.4770790047389427325.stgit@dwillia2-desk3.amr.corp.intel.com
State New
Headers show
Series
  • [v5] mm/memory_hotplug: Fix remove_memory() lockdep splat
Related show

Commit Message

Dan Williams Jan. 25, 2020, 1:07 a.m. UTC
The daxctl unit test for the dax_kmem driver currently triggers the
(false positive) lockdep splat below. It results from the fact that
remove_memory_block_devices() is invoked under the mem_hotplug_lock()
causing lockdep entanglements with cpu_hotplug_lock() and sysfs (kernfs
active state tracking). It is a false positive because the sysfs
attribute path triggering the memory remove is not the same attribute
path associated with memory-block device.

sysfs_break_active_protection() is not applicable since there is no real
deadlock conflict, instead move memory-block device removal outside the
lock. The mem_hotplug_lock() is not needed to synchronize the
memory-block device removal vs the page online state, that is already
handled by lock_device_hotplug(). Specifically, lock_device_hotplug() is
sufficient to allow try_remove_memory() to check the offline state of
the memblocks and be assured that any in progress online attempts are
flushed / blocked by kernfs_drain() / attribute removal.

The add_memory() path safely creates memblock devices under the
mem_hotplug_lock(). There is no kernfs active state synchronization in
the memblock device_register() path, so nothing to fix there.

This change is only possible thanks to the recent change that refactored
memory block device removal out of arch_remove_memory() (commit
4c4b7f9ba948 mm/memory_hotplug: remove memory block devices before
arch_remove_memory()), and David's due diligence tracking down the
guarantees afforded by kernfs_drain(). Not flagged for -stable since
this only impacts ongoing development and lockdep validation, not a
runtime issue.

    ======================================================
    WARNING: possible circular locking dependency detected
    5.5.0-rc3+ #230 Tainted: G           OE
    ------------------------------------------------------
    lt-daxctl/6459 is trying to acquire lock:
    ffff99c7f0003510 (kn->count#241){++++}, at: kernfs_remove_by_name_ns+0x41/0x80

    but task is already holding lock:
    ffffffffa76a5450 (mem_hotplug_lock.rw_sem){++++}, at: percpu_down_write+0x20/0xe0

    which lock already depends on the new lock.


    the existing dependency chain (in reverse order) is:

    -> #2 (mem_hotplug_lock.rw_sem){++++}:
           __lock_acquire+0x39c/0x790
           lock_acquire+0xa2/0x1b0
           get_online_mems+0x3e/0xb0
           kmem_cache_create_usercopy+0x2e/0x260
           kmem_cache_create+0x12/0x20
           ptlock_cache_init+0x20/0x28
           start_kernel+0x243/0x547
           secondary_startup_64+0xb6/0xc0

    -> #1 (cpu_hotplug_lock.rw_sem){++++}:
           __lock_acquire+0x39c/0x790
           lock_acquire+0xa2/0x1b0
           cpus_read_lock+0x3e/0xb0
           online_pages+0x37/0x300
           memory_subsys_online+0x17d/0x1c0
           device_online+0x60/0x80
           state_store+0x65/0xd0
           kernfs_fop_write+0xcf/0x1c0
           vfs_write+0xdb/0x1d0
           ksys_write+0x65/0xe0
           do_syscall_64+0x5c/0xa0
           entry_SYSCALL_64_after_hwframe+0x49/0xbe

    -> #0 (kn->count#241){++++}:
           check_prev_add+0x98/0xa40
           validate_chain+0x576/0x860
           __lock_acquire+0x39c/0x790
           lock_acquire+0xa2/0x1b0
           __kernfs_remove+0x25f/0x2e0
           kernfs_remove_by_name_ns+0x41/0x80
           remove_files.isra.0+0x30/0x70
           sysfs_remove_group+0x3d/0x80
           sysfs_remove_groups+0x29/0x40
           device_remove_attrs+0x39/0x70
           device_del+0x16a/0x3f0
           device_unregister+0x16/0x60
           remove_memory_block_devices+0x82/0xb0
           try_remove_memory+0xb5/0x130
           remove_memory+0x26/0x40
           dev_dax_kmem_remove+0x44/0x6a [kmem]
           device_release_driver_internal+0xe4/0x1c0
           unbind_store+0xef/0x120
           kernfs_fop_write+0xcf/0x1c0
           vfs_write+0xdb/0x1d0
           ksys_write+0x65/0xe0
           do_syscall_64+0x5c/0xa0
           entry_SYSCALL_64_after_hwframe+0x49/0xbe

    other info that might help us debug this:

    Chain exists of:
      kn->count#241 --> cpu_hotplug_lock.rw_sem --> mem_hotplug_lock.rw_sem

     Possible unsafe locking scenario:

           CPU0                    CPU1
           ----                    ----
      lock(mem_hotplug_lock.rw_sem);
                                   lock(cpu_hotplug_lock.rw_sem);
                                   lock(mem_hotplug_lock.rw_sem);
      lock(kn->count#241);

     *** DEADLOCK ***

No fixes tag as this has been a long standing issue that predated the
addition of kernfs lockdep annotations.

Cc: Vishal Verma <vishal.l.verma@intel.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
Changes since v4 [1]:
- Drop the unnecessary consideration of mem->section_count.
  kernfs_drain() + lock_device_hotplug() is sufficient protection
  (David)

[1]: http://lore.kernel.org/r/157869128062.2451572.4093315441083744888.stgit@dwillia2-desk3.amr.corp.intel.com

 mm/memory_hotplug.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

Comments

David Hildenbrand Jan. 27, 2020, 7:44 a.m. UTC | #1
> Am 25.01.2020 um 02:23 schrieb Dan Williams <dan.j.williams@intel.com>:
> 
> ´╗┐The daxctl unit test for the dax_kmem driver currently triggers the
> (false positive) lockdep splat below. It results from the fact that
> remove_memory_block_devices() is invoked under the mem_hotplug_lock()
> causing lockdep entanglements with cpu_hotplug_lock() and sysfs (kernfs
> active state tracking). It is a false positive because the sysfs
> attribute path triggering the memory remove is not the same attribute
> path associated with memory-block device.
> 
> sysfs_break_active_protection() is not applicable since there is no real
> deadlock conflict, instead move memory-block device removal outside the
> lock. The mem_hotplug_lock() is not needed to synchronize the
> memory-block device removal vs the page online state, that is already
> handled by lock_device_hotplug(). Specifically, lock_device_hotplug() is
> sufficient to allow try_remove_memory() to check the offline state of
> the memblocks and be assured that any in progress online attempts are
> flushed / blocked by kernfs_drain() / attribute removal.
> 
> The add_memory() path safely creates memblock devices under the
> mem_hotplug_lock(). There is no kernfs active state synchronization in
> the memblock device_register() path, so nothing to fix there.
> 
> This change is only possible thanks to the recent change that refactored
> memory block device removal out of arch_remove_memory() (commit
> 4c4b7f9ba948 mm/memory_hotplug: remove memory block devices before
> arch_remove_memory()), and David's due diligence tracking down the
> guarantees afforded by kernfs_drain(). Not flagged for -stable since
> this only impacts ongoing development and lockdep validation, not a
> runtime issue.
> 
>    ======================================================
>    WARNING: possible circular locking dependency detected
>    5.5.0-rc3+ #230 Tainted: G           OE
>    ------------------------------------------------------
>    lt-daxctl/6459 is trying to acquire lock:
>    ffff99c7f0003510 (kn->count#241){++++}, at: kernfs_remove_by_name_ns+0x41/0x80
> 
>    but task is already holding lock:
>    ffffffffa76a5450 (mem_hotplug_lock.rw_sem){++++}, at: percpu_down_write+0x20/0xe0
> 
>    which lock already depends on the new lock.
> 
> 
>    the existing dependency chain (in reverse order) is:
> 
>    -> #2 (mem_hotplug_lock.rw_sem){++++}:
>           __lock_acquire+0x39c/0x790
>           lock_acquire+0xa2/0x1b0
>           get_online_mems+0x3e/0xb0
>           kmem_cache_create_usercopy+0x2e/0x260
>           kmem_cache_create+0x12/0x20
>           ptlock_cache_init+0x20/0x28
>           start_kernel+0x243/0x547
>           secondary_startup_64+0xb6/0xc0
> 
>    -> #1 (cpu_hotplug_lock.rw_sem){++++}:
>           __lock_acquire+0x39c/0x790
>           lock_acquire+0xa2/0x1b0
>           cpus_read_lock+0x3e/0xb0
>           online_pages+0x37/0x300
>           memory_subsys_online+0x17d/0x1c0
>           device_online+0x60/0x80
>           state_store+0x65/0xd0
>           kernfs_fop_write+0xcf/0x1c0
>           vfs_write+0xdb/0x1d0
>           ksys_write+0x65/0xe0
>           do_syscall_64+0x5c/0xa0
>           entry_SYSCALL_64_after_hwframe+0x49/0xbe
> 
>    -> #0 (kn->count#241){++++}:
>           check_prev_add+0x98/0xa40
>           validate_chain+0x576/0x860
>           __lock_acquire+0x39c/0x790
>           lock_acquire+0xa2/0x1b0
>           __kernfs_remove+0x25f/0x2e0
>           kernfs_remove_by_name_ns+0x41/0x80
>           remove_files.isra.0+0x30/0x70
>           sysfs_remove_group+0x3d/0x80
>           sysfs_remove_groups+0x29/0x40
>           device_remove_attrs+0x39/0x70
>           device_del+0x16a/0x3f0
>           device_unregister+0x16/0x60
>           remove_memory_block_devices+0x82/0xb0
>           try_remove_memory+0xb5/0x130
>           remove_memory+0x26/0x40
>           dev_dax_kmem_remove+0x44/0x6a [kmem]
>           device_release_driver_internal+0xe4/0x1c0
>           unbind_store+0xef/0x120
>           kernfs_fop_write+0xcf/0x1c0
>           vfs_write+0xdb/0x1d0
>           ksys_write+0x65/0xe0
>           do_syscall_64+0x5c/0xa0
>           entry_SYSCALL_64_after_hwframe+0x49/0xbe
> 
>    other info that might help us debug this:
> 
>    Chain exists of:
>      kn->count#241 --> cpu_hotplug_lock.rw_sem --> mem_hotplug_lock.rw_sem
> 
>     Possible unsafe locking scenario:
> 
>           CPU0                    CPU1
>           ----                    ----
>      lock(mem_hotplug_lock.rw_sem);
>                                   lock(cpu_hotplug_lock.rw_sem);
>                                   lock(mem_hotplug_lock.rw_sem);
>      lock(kn->count#241);
> 
>     *** DEADLOCK ***
> 
> No fixes tag as this has been a long standing issue that predated the
> addition of kernfs lockdep annotations.
> 
> Cc: Vishal Verma <vishal.l.verma@intel.com>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> ---
> Changes since v4 [1]:
> - Drop the unnecessary consideration of mem->section_count.
>  kernfs_drain() + lock_device_hotplug() is sufficient protection
>  (David)
> 
> [1]: http://lore.kernel.org/r/157869128062.2451572.4093315441083744888.stgit@dwillia2-desk3.amr.corp.intel.com
> 
> mm/memory_hotplug.c |    9 ++++++---
> 1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 55ac23ef11c1..65ddaf3a2a12 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1763,8 +1763,6 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
> 
>    BUG_ON(check_hotplug_memory_range(start, size));
> 
> -    mem_hotplug_begin();
> -
>    /*
>     * All memory blocks must be offlined before removing memory.  Check
>     * whether all memory blocks in question are offline and return error
> @@ -1777,9 +1775,14 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
>    /* remove memmap entry */
>    firmware_map_remove(start, start + size, "System RAM");
> 
> -    /* remove memory block devices before removing memory */
> +    /*
> +     * Memory block device removal under the device_hotplug_lock is
> +     * a barrier against racing online attempts.
> +     */
>    remove_memory_block_devices(start, size);
> 
> +    mem_hotplug_begin();
> +
>    arch_remove_memory(nid, start, size, NULL);
>    memblock_free(start, size);
>    memblock_remove(start, size);
> 
> 

Thanks!

Reviewed-by: David Hildenbrand <david@redhat.com>
Michal Hocko Jan. 27, 2020, 1:41 p.m. UTC | #2
On Fri 24-01-20 17:07:21, Dan Williams wrote:
> The daxctl unit test for the dax_kmem driver currently triggers the
> (false positive) lockdep splat below. It results from the fact that
> remove_memory_block_devices() is invoked under the mem_hotplug_lock()
> causing lockdep entanglements with cpu_hotplug_lock() and sysfs (kernfs
> active state tracking). It is a false positive because the sysfs
> attribute path triggering the memory remove is not the same attribute
> path associated with memory-block device.
> 
> sysfs_break_active_protection() is not applicable since there is no real
> deadlock conflict, instead move memory-block device removal outside the
> lock. The mem_hotplug_lock() is not needed to synchronize the
> memory-block device removal vs the page online state, that is already
> handled by lock_device_hotplug(). Specifically, lock_device_hotplug() is
> sufficient to allow try_remove_memory() to check the offline state of
> the memblocks and be assured that any in progress online attempts are
> flushed / blocked by kernfs_drain() / attribute removal.
> 
> The add_memory() path safely creates memblock devices under the
> mem_hotplug_lock(). There is no kernfs active state synchronization in
> the memblock device_register() path, so nothing to fix there.
> 
> This change is only possible thanks to the recent change that refactored
> memory block device removal out of arch_remove_memory() (commit
> 4c4b7f9ba948 mm/memory_hotplug: remove memory block devices before
> arch_remove_memory()), and David's due diligence tracking down the
> guarantees afforded by kernfs_drain(). Not flagged for -stable since
> this only impacts ongoing development and lockdep validation, not a
> runtime issue.
> 
>     ======================================================
>     WARNING: possible circular locking dependency detected
>     5.5.0-rc3+ #230 Tainted: G           OE
>     ------------------------------------------------------
>     lt-daxctl/6459 is trying to acquire lock:
>     ffff99c7f0003510 (kn->count#241){++++}, at: kernfs_remove_by_name_ns+0x41/0x80
> 
>     but task is already holding lock:
>     ffffffffa76a5450 (mem_hotplug_lock.rw_sem){++++}, at: percpu_down_write+0x20/0xe0
> 
>     which lock already depends on the new lock.
> 
> 
>     the existing dependency chain (in reverse order) is:
> 
>     -> #2 (mem_hotplug_lock.rw_sem){++++}:
>            __lock_acquire+0x39c/0x790
>            lock_acquire+0xa2/0x1b0
>            get_online_mems+0x3e/0xb0
>            kmem_cache_create_usercopy+0x2e/0x260
>            kmem_cache_create+0x12/0x20
>            ptlock_cache_init+0x20/0x28
>            start_kernel+0x243/0x547
>            secondary_startup_64+0xb6/0xc0
> 
>     -> #1 (cpu_hotplug_lock.rw_sem){++++}:
>            __lock_acquire+0x39c/0x790
>            lock_acquire+0xa2/0x1b0
>            cpus_read_lock+0x3e/0xb0
>            online_pages+0x37/0x300
>            memory_subsys_online+0x17d/0x1c0
>            device_online+0x60/0x80
>            state_store+0x65/0xd0
>            kernfs_fop_write+0xcf/0x1c0
>            vfs_write+0xdb/0x1d0
>            ksys_write+0x65/0xe0
>            do_syscall_64+0x5c/0xa0
>            entry_SYSCALL_64_after_hwframe+0x49/0xbe
> 
>     -> #0 (kn->count#241){++++}:
>            check_prev_add+0x98/0xa40
>            validate_chain+0x576/0x860
>            __lock_acquire+0x39c/0x790
>            lock_acquire+0xa2/0x1b0
>            __kernfs_remove+0x25f/0x2e0
>            kernfs_remove_by_name_ns+0x41/0x80
>            remove_files.isra.0+0x30/0x70
>            sysfs_remove_group+0x3d/0x80
>            sysfs_remove_groups+0x29/0x40
>            device_remove_attrs+0x39/0x70
>            device_del+0x16a/0x3f0
>            device_unregister+0x16/0x60
>            remove_memory_block_devices+0x82/0xb0
>            try_remove_memory+0xb5/0x130
>            remove_memory+0x26/0x40
>            dev_dax_kmem_remove+0x44/0x6a [kmem]
>            device_release_driver_internal+0xe4/0x1c0
>            unbind_store+0xef/0x120
>            kernfs_fop_write+0xcf/0x1c0
>            vfs_write+0xdb/0x1d0
>            ksys_write+0x65/0xe0
>            do_syscall_64+0x5c/0xa0
>            entry_SYSCALL_64_after_hwframe+0x49/0xbe
> 
>     other info that might help us debug this:
> 
>     Chain exists of:
>       kn->count#241 --> cpu_hotplug_lock.rw_sem --> mem_hotplug_lock.rw_sem
> 
>      Possible unsafe locking scenario:
> 
>            CPU0                    CPU1
>            ----                    ----
>       lock(mem_hotplug_lock.rw_sem);
>                                    lock(cpu_hotplug_lock.rw_sem);
>                                    lock(mem_hotplug_lock.rw_sem);
>       lock(kn->count#241);
> 
>      *** DEADLOCK ***
> 
> No fixes tag as this has been a long standing issue that predated the
> addition of kernfs lockdep annotations.
> 
> Cc: Vishal Verma <vishal.l.verma@intel.com>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Pavel Tatashin <pasha.tatashin@soleen.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Dave Hansen <dave.hansen@linux.intel.com>
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>

Acked-by: Michal Hocko <mhocko@suse.com>
Thanks!

> ---
> Changes since v4 [1]:
> - Drop the unnecessary consideration of mem->section_count.
>   kernfs_drain() + lock_device_hotplug() is sufficient protection
>   (David)
> 
> [1]: http://lore.kernel.org/r/157869128062.2451572.4093315441083744888.stgit@dwillia2-desk3.amr.corp.intel.com
> 
>  mm/memory_hotplug.c |    9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 55ac23ef11c1..65ddaf3a2a12 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1763,8 +1763,6 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
>  
>  	BUG_ON(check_hotplug_memory_range(start, size));
>  
> -	mem_hotplug_begin();
> -
>  	/*
>  	 * All memory blocks must be offlined before removing memory.  Check
>  	 * whether all memory blocks in question are offline and return error
> @@ -1777,9 +1775,14 @@ static int __ref try_remove_memory(int nid, u64 start, u64 size)
>  	/* remove memmap entry */
>  	firmware_map_remove(start, start + size, "System RAM");
>  
> -	/* remove memory block devices before removing memory */
> +	/*
> +	 * Memory block device removal under the device_hotplug_lock is
> +	 * a barrier against racing online attempts.
> +	 */
>  	remove_memory_block_devices(start, size);
>  
> +	mem_hotplug_begin();
> +
>  	arch_remove_memory(nid, start, size, NULL);
>  	memblock_free(start, size);
>  	memblock_remove(start, size);

Patch
diff mbox series

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 55ac23ef11c1..65ddaf3a2a12 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1763,8 +1763,6 @@  static int __ref try_remove_memory(int nid, u64 start, u64 size)
 
 	BUG_ON(check_hotplug_memory_range(start, size));
 
-	mem_hotplug_begin();
-
 	/*
 	 * All memory blocks must be offlined before removing memory.  Check
 	 * whether all memory blocks in question are offline and return error
@@ -1777,9 +1775,14 @@  static int __ref try_remove_memory(int nid, u64 start, u64 size)
 	/* remove memmap entry */
 	firmware_map_remove(start, start + size, "System RAM");
 
-	/* remove memory block devices before removing memory */
+	/*
+	 * Memory block device removal under the device_hotplug_lock is
+	 * a barrier against racing online attempts.
+	 */
 	remove_memory_block_devices(start, size);
 
+	mem_hotplug_begin();
+
 	arch_remove_memory(nid, start, size, NULL);
 	memblock_free(start, size);
 	memblock_remove(start, size);