| Message ID | 0be1a0a03367f7230497a2e7a5ed47d2a2d5ae1a.1611809091.git.baolin.wang@linux.alibaba.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v3] blk-cgroup: Use cond_resched() when destroy blkgs | expand |
On 1/27/21 10:58 PM, Baolin Wang wrote: > On !PREEMPT kernel, we can get below softlockup when doing stress > testing with creating and destroying block cgroup repeatly. The > reason is it may take a long time to acquire the queue's lock in > the loop of blkcg_destroy_blkgs(), or the system can accumulate a > huge number of blkgs in pathological cases. We can add a need_resched() > check on each loop and release locks and do cond_resched() if true > to avoid this issue, since the blkcg_destroy_blkgs() is not called > from atomic contexts. > > [ 4757.010308] watchdog: BUG: soft lockup - CPU#11 stuck for 94s! > [ 4757.010698] Call trace: > [ 4757.010700] blkcg_destroy_blkgs+0x68/0x150 > [ 4757.010701] cgwb_release_workfn+0x104/0x158 > [ 4757.010702] process_one_work+0x1bc/0x3f0 > [ 4757.010704] worker_thread+0x164/0x468 > [ 4757.010705] kthread+0x108/0x138 Thanks, applied.
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 02ce205..4b4fcb5 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1016,6 +1016,8 @@ static void blkcg_css_offline(struct cgroup_subsys_state *css) */ void blkcg_destroy_blkgs(struct blkcg *blkcg) { + might_sleep(); + spin_lock_irq(&blkcg->lock); while (!hlist_empty(&blkcg->blkg_list)) { @@ -1023,14 +1025,20 @@ void blkcg_destroy_blkgs(struct blkcg *blkcg) struct blkcg_gq, blkcg_node); struct request_queue *q = blkg->q; - if (spin_trylock(&q->queue_lock)) { - blkg_destroy(blkg); - spin_unlock(&q->queue_lock); - } else { + if (need_resched() || !spin_trylock(&q->queue_lock)) { + /* + * Given that the system can accumulate a huge number + * of blkgs in pathological cases, check to see if we + * need to rescheduling to avoid softlockup. + */ spin_unlock_irq(&blkcg->lock); - cpu_relax(); + cond_resched(); spin_lock_irq(&blkcg->lock); + continue; } + + blkg_destroy(blkg); + spin_unlock(&q->queue_lock); } spin_unlock_irq(&blkcg->lock);
On !PREEMPT kernel, we can get below softlockup when doing stress testing with creating and destroying block cgroup repeatly. The reason is it may take a long time to acquire the queue's lock in the loop of blkcg_destroy_blkgs(), or the system can accumulate a huge number of blkgs in pathological cases. We can add a need_resched() check on each loop and release locks and do cond_resched() if true to avoid this issue, since the blkcg_destroy_blkgs() is not called from atomic contexts. [ 4757.010308] watchdog: BUG: soft lockup - CPU#11 stuck for 94s! [ 4757.010698] Call trace: [ 4757.010700] blkcg_destroy_blkgs+0x68/0x150 [ 4757.010701] cgwb_release_workfn+0x104/0x158 [ 4757.010702] process_one_work+0x1bc/0x3f0 [ 4757.010704] worker_thread+0x164/0x468 [ 4757.010705] kthread+0x108/0x138 Suggested-by: Tejun Heo <tj@kernel.org> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com> --- Changes from v2: - Simplify logics suggested by Jens. Changes from v1: - Add might_sleep() in blkcg_destroy_blkgs(). - Add an explicitly need_resched() check before releasing lock. - Add some comments. --- block/blk-cgroup.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)