| Message ID | 1503086474-20187-1-git-send-email-longman@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Fri, 2017-08-18 at 16:01 -0400, Waiman Long wrote: > [ ... ] > Instead of using bd_mutex, a new global blktrace mutex is now used > to protect against concurrent access, creation and destruction of the > blk_trace structure that is used only in the blktrace.c file. As > blktrace files will not be frequently accessed, using a global mutex > should not cause any performance problem. > [ ... ] > +/* > + * The bd_mutex was used previously for protecting blk_trace structure. > + * That could lead to deadlock with concurrent block device deletion and > + * sysfs access. So a global blktrace_mutex is now used instead for > + * protecting the blk_trace structure. > + * > + * The references to the opened sysfs or device files should prevent the > + * underlying block device from being removed. > + */ > +static DEFINE_MUTEX(blktrace_mutex); Hello Waiman, Thanks for having addressed my previous comment. Regarding this patch: sorry but I don't think it's a good idea to use a global mutex for serializing accesses to tracing data of a single block device. Global mutexes create unwanted lock dependencies between different block devices. Additionally, on multiprocessor systems global mutexes can cause cache line ping-pong between processors and hence can cause a severe slowdown. Please make blktrace_mutex per block device instead of global. Thanks, Bart.
On 08/18/2017 04:18 PM, Bart Van Assche wrote: > On Fri, 2017-08-18 at 16:01 -0400, Waiman Long wrote: >> [ ... ] >> Instead of using bd_mutex, a new global blktrace mutex is now used >> to protect against concurrent access, creation and destruction of the >> blk_trace structure that is used only in the blktrace.c file. As >> blktrace files will not be frequently accessed, using a global mutex >> should not cause any performance problem. >> [ ... ] >> +/* >> + * The bd_mutex was used previously for protecting blk_trace structure. >> + * That could lead to deadlock with concurrent block device deletion and >> + * sysfs access. So a global blktrace_mutex is now used instead for >> + * protecting the blk_trace structure. >> + * >> + * The references to the opened sysfs or device files should prevent the >> + * underlying block device from being removed. >> + */ >> +static DEFINE_MUTEX(blktrace_mutex); > Hello Waiman, > > Thanks for having addressed my previous comment. Regarding this patch: sorry > but I don't think it's a good idea to use a global mutex for serializing > accesses to tracing data of a single block device. Global mutexes create > unwanted lock dependencies between different block devices. Additionally, on > multiprocessor systems global mutexes can cause cache line ping-pong between > processors and hence can cause a severe slowdown. Please make blktrace_mutex > per block device instead of global. I fully understand the problem of a global lock. The main reason of using a global lock here is that the blktrace APIs are not in a performance critical path. In fact, I think it is used primarily for debugging purpose. Activating it will certainly slow thing down no matter what kind of lock is used. I also don't believe that the blktrace APIs will be used in a high enough frequency that it will cause a performance issue. Please let me know if I am wrong in my assumptions. Cheers, Longman
diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c index bc364f8..ec5a919 100644 --- a/kernel/trace/blktrace.c +++ b/kernel/trace/blktrace.c @@ -624,6 +624,17 @@ int blk_trace_startstop(struct request_queue *q, int start) } EXPORT_SYMBOL_GPL(blk_trace_startstop); +/* + * The bd_mutex was used previously for protecting blk_trace structure. + * That could lead to deadlock with concurrent block device deletion and + * sysfs access. So a global blktrace_mutex is now used instead for + * protecting the blk_trace structure. + * + * The references to the opened sysfs or device files should prevent the + * underlying block device from being removed. + */ +static DEFINE_MUTEX(blktrace_mutex); + /** * blk_trace_ioctl: - handle the ioctls associated with tracing * @bdev: the block device @@ -641,7 +652,7 @@ int blk_trace_ioctl(struct block_device *bdev, unsigned cmd, char __user *arg) if (!q) return -ENXIO; - mutex_lock(&bdev->bd_mutex); + mutex_lock(&blktrace_mutex); switch (cmd) { case BLKTRACESETUP: @@ -667,7 +678,7 @@ int blk_trace_ioctl(struct block_device *bdev, unsigned cmd, char __user *arg) break; } - mutex_unlock(&bdev->bd_mutex); + mutex_unlock(&blktrace_mutex); return ret; } @@ -1622,7 +1633,7 @@ static ssize_t sysfs_blk_trace_attr_show(struct device *dev, if (q == NULL) goto out_bdput; - mutex_lock(&bdev->bd_mutex); + mutex_lock(&blktrace_mutex); if (attr == &dev_attr_enable) { ret = sprintf(buf, "%u\n", !!q->blk_trace); @@ -1641,7 +1652,7 @@ static ssize_t sysfs_blk_trace_attr_show(struct device *dev, ret = sprintf(buf, "%llu\n", q->blk_trace->end_lba); out_unlock_bdev: - mutex_unlock(&bdev->bd_mutex); + mutex_unlock(&blktrace_mutex); out_bdput: bdput(bdev); out: @@ -1683,7 +1694,7 @@ static ssize_t sysfs_blk_trace_attr_store(struct device *dev, if (q == NULL) goto out_bdput; - mutex_lock(&bdev->bd_mutex); + mutex_lock(&blktrace_mutex); if (attr == &dev_attr_enable) { if (value) @@ -1709,7 +1720,7 @@ static ssize_t sysfs_blk_trace_attr_store(struct device *dev, } out_unlock_bdev: - mutex_unlock(&bdev->bd_mutex); + mutex_unlock(&blktrace_mutex); out_bdput: bdput(bdev); out:
The lockdep code had reported the following unsafe locking scenario: CPU0 CPU1 ---- ---- lock(s_active#228); lock(&bdev->bd_mutex/1); lock(s_active#228); lock(&bdev->bd_mutex); *** DEADLOCK *** The deadlock may happen when one task (CPU1) is trying to delete a partition in a block device and another task (CPU0) is accessing tracing sysfs file (e.g. /sys/block/dm-1/trace/act_mask) in that partition. The s_active isn't an actual lock. It is a reference count (kn->count) on the sysfs (kernfs) file. Removal of a sysfs file, however, require a wait until all the references are gone. The reference count is treated like a rwsem using lockdep instrumentation code. The fact that a thread is in the sysfs callback method or in the ioctl call means there is a reference to the opended sysfs or device file. That should prevent the underlying block structure from being removed. Instead of using bd_mutex, a new global blktrace mutex is now used to protect against concurrent access, creation and destruction of the blk_trace structure that is used only in the blktrace.c file. As blktrace files will not be frequently accessed, using a global mutex should not cause any performance problem. Signed-off-by: Waiman Long <longman@redhat.com> --- v4: - Use blktrace_mutex in blk_trace_ioctl() as well. v3: - Use a global blktrace_mutex to serialize sysfs attribute accesses instead of the bd_mutex. v2: - Use READ_ONCE() and smp_store_mb() to read and write bd_deleting. - Check for signal in the mutex_trylock loops. - Use usleep() instead of schedule() for RT tasks. kernel/trace/blktrace.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)